Interpret bytes to numpy

The way to interpret bytes to numpy array.
In this post, I'm gonna use MNIST data you can down load from this link 'http://yann.lecun.com/exdb/mnist/'. At first, data format of label data of MNIST should be clarified.It's written on 'http://yann.lecun.com/exdb/mnist/' as bellow.

    
[offset] [type]          [value]          [description] 
0000     32 bit integer  0x00000801(2049) magic number (MSB first) 
0004     32 bit integer  60000            number of items 
0008     unsigned byte   ??               label 
0009     unsigned byte   ??               label 
........ 
xxxx     unsigned byte   ??               label
    

Get contents as bytes.

    
In [1]: import gzip

In [2]: train_label_file = 'train-labels-idx1-ubyte.gz'

In [3]: label_gzip = gzip.open(train_label_file)

In [4]: label_contents = label_gzip.read()

In [5]: label_gzip.close()
    

Now, you can interpret a buffer as a 1-dimensional array with 'frombuffer' function offered by numpy. However, You gotta be careful for dtype argument of frombuffer function. As a default, 'frombuffer' function try to interpret buffer as float which is 64-bit. However from the above information, labels are 1 byte data. Therefor using uint-8 is appropriate. As a trial, compare both numpy.

    
In [6]: # check contents

In [7]: label_contents[8:32]
Out[7]: b'\x05\x00\x04\x01\t\x02\x01\x03\x01\x04\x03\x05\x03\x06\x01\x07\x02\x08\x06\t\x04\x00\t\x01'
In [8]: # invoke frombuffer with float.

In [9]: float_array = np.frombuffer(label_contents,offset=8)

In [10]: # invoke frombuffer with uint8

In [11]: uint8_array = np.frombuffer(label_contents, dtype='uint8',offset=8)

In [12]: # check the value of float_array

In [13]: float_array
Out[13]: 
array([  3.32878858e-294,   6.14613936e-275,   1.13924062e-303, ...,
         6.57718904e-299,   7.47547348e-299,   5.21006186e-270])

In [14]: # check the value of uint8

In [25]: uint8_array
Out[20]: array([5, 0, 4, ..., 5, 6, 8], dtype=uint8)

In [26]: # check the number of each array

In [27]: len(float_array)
Out[27]: 7500

In [28]: len(uint8_array)
Out[28]: 60000

In [29]: 
    

From the result above, 'frombuffer' with float retrieve value per 64bit from buffer, whereas with uint8 retrieve value per 8 bit from buffer. (We can see these from the number of elements of each array. The number of elements of float is one forth of the one of uint8:))

How to configure IPsecVPN of Vyatta5600 on SoftLayer

The protocl of IPsecVPN is sort of complicated and perplexing for me. As such, this is somewhat of notes of how to configure IPsecVPN of Vyatta5600 on SoftLayer.
Given that you have two Vyatta5600, one is TOK02, the other is SNG02 as shown by the figure 1 bellow. I'm gonna show how to establish site-to-site IPsecVPN between 'tokvyatta' and 'sngvyatta'.

figure 1.

Note : All subnet and IP address are provisional, different from ones I used. Providing that you configure IPsecVPN based on this article, you should replace them as appropriate.

0. Prerequisite

Before getting started, there a few things you should do preliminarily.

• Both data plane interface used for IPsec VPN must be configured already. In my case, both interfaces are configured as 'dp0bond1'.
  
  Note : These IP addresses are provisional, different from ones I used.
  
  tokvyatta
  

  
          set interface bonding dp0bond1 address '161.202.0.1/27'
          

  
  
  sngvyatta

  
          set interface bonding dp0bond1 address '119.81.0.1/27'
          

  
  
• Subnet to be configured as local/remote 'prefix' should be associated with Vyatta beforehand.

1. Set the IKE group for phase 1

First of all, you can configure parameter used for IKE phase1 negotiation. I configured it as 'ESP-1E'.
In my case, I configure parameters as followings.

• SHA-1 : Cipher to encrypt packets during IKE phase 1
• AES-256 : Hash argorithm to authenticate during IKE phase 1

Actually, you can configure multiple set of these parameter as "proposal". In my case, I only configure one proposal called "proposal 1".
And you might wanna set lifetime which is the duration of the ISAKKP SA. If lifetime of the ISAKMP SA expires, a new Phase 1 negotiation will take place. Then new encryption, hash, and keying imformation is established in a new pair of ISAKMP SAs. In this case, lifetime is set to '3600'

tokvyatta

 
set security vpn ipsec ike-group IKE-1E lifetime '3600'
set security vpn ipsec ike-group IKE-1E proposal 1 dh-group '2'
set security vpn ipsec ike-group IKE-1E proposal 1 encryption 'aes256'
set security vpn ipsec ike-group IKE-1E proposal 1 hash 'sha1'

sngvyatta

set security vpn ipsec ike-group IKE-1E lifetime '3600'
set security vpn ipsec ike-group IKE-1E proposal 1 dh-group '2'
set security vpn ipsec ike-group IKE-1E proposal 1 encryption 'aes256'
set security vpn ipsec ike-group IKE-1E proposal 1 hash 'sha1'

2. Configure ESP group

ESP, Encapsulated Security Payload is a protocol which provides authentication and encryption for IP packet. In my case, I configured parameter as following.

• AES-256 : Cipher to encrypt user data across the IPsec tunnel
• SHA-1 : Hashing function to authenticate packets in the IPsec tunnel

In the same manner as IKE group, you are allowed to set several set of these parameter.　You can also set lifetime of ESP group. I'll use 1800 for that.

tokvyatta

set security vpn ipsec esp-group ESP-1E compression 'disable'
set security vpn ipsec esp-group ESP-1E lifetime '1800'
set security vpn ipsec esp-group ESP-1E mode 'tunnel'
set security vpn ipsec esp-group ESP-1E pfs 'enable'
set security vpn ipsec esp-group ESP-1E proposal 1 encryption 'aes256'
set security vpn ipsec esp-group ESP-1E proposal 1 hash 'sha1'

sngvyatta

set security vpn ipsec esp-group ESP-1E compression 'disable'
set security vpn ipsec esp-group ESP-1E lifetime '1800'
set security vpn ipsec esp-group ESP-1E mode 'tunnel'
set security vpn ipsec esp-group ESP-1E pfs 'enable'
set security vpn ipsec esp-group ESP-1E proposal 1 encryption 'aes256'
set security vpn ipsec esp-group ESP-1E proposal 1 hash 'sha1'

3. Define site-to-site connection

Here, you are supposed to set plenty of parameter as bellow.　But hang in there, Every cloud has a silver lining.

• IP address of the remote router.
• Authentication mode that the peers use to authenticate one another. Apparently, Vyatta5600 supports 'pre-shared-secret', 'rsa', 'x509'. Incidentally, you can check these with command line interface as bellow.

  
  vyatta@tokvyatta1# set security vpn ipsec site-to-site peer 119.81.0.1 authentication mode 
  Possible Completions:
     pre-shared-secret Use pre-shared secret key
     rsa               Use RSA key
     x509              Use X.509 certificate
        
  [edit]
  vyatta@tokvyatta1#
                        

  In my case, I will use "pre-shared secret key(PSK)", which means The secret key which is character string should be agreed upon beforehand by both parties. Note that 'pre-shared secret' is not a "password". It will be utilized to generate hashed key to form a fingerprint that proves the identity of each end point. Therefore, the longer and more complex character strings is, the more secure it become and prevent from being compromised. In this example, I'll use 'secret-string' which is not good example.
• IKE group which you can select from what you configured in 1. Set the IKE group for phase 1
• ESP-group wihich you can selecvt from what you configured in 2. Configure ESP group

tokvyatta

set security vpn ipsec site-to-site peer 119.81.00.1 authentication mode 'pre-shared-secret'
set security vpn ipsec site-to-site peer 119.81.00.1 authentication pre-shared-secret 'secret-string'
set security vpn ipsec site-to-site peer 119.81.00.1 connection-type 'initiate'
set security vpn ipsec site-to-site peer 119.81.00.1 default-esp-group 'ESP-1E'
set security vpn ipsec site-to-site peer 119.81.00.1 ike-group 'IKE-1E'
set security vpn ipsec site-to-site peer 119.81.00.1 local-address '161.202.00.1'
set security vpn ipsec site-to-site peer 119.81.00.1 tunnel 1 allow-nat-networks 'disable'
set security vpn ipsec site-to-site peer 119.81.00.1 tunnel 1 allow-public-networks 'disable'
set security vpn ipsec site-to-site peer 119.81.00.1 tunnel 1 local prefix '161.202.60.0/29'
set security vpn ipsec site-to-site peer 119.81.00.1 tunnel 1 remote prefix '119.81.40.0/29'

sngvyatta

set security vpn ipsec site-to-site peer 161.202.0.1 authentication mode 'pre-shared-secret'
set security vpn ipsec site-to-site peer 161.202.0.1 authentication pre-shared-secret 'secret-string'
set security vpn ipsec site-to-site peer 161.202.0.1 connection-type 'initiate'
set security vpn ipsec site-to-site peer 161.202.0.1 default-esp-group 'ESP-1E'
set security vpn ipsec site-to-site peer 161.202.0.1 ike-group 'IKE-1E'
set security vpn ipsec site-to-site peer 161.202.0.1 local-address '119.81.77.189'
set security vpn ipsec site-to-site peer 161.202.0.1 tunnel 1 allow-nat-networks 'disable'
set security vpn ipsec site-to-site peer 161.202.0.1 tunnel 1 allow-public-networks 'disable'
set security vpn ipsec site-to-site peer 161.202.0.1 tunnel 1 local prefix '119.81.40.0/29'
set security vpn ipsec site-to-site peer 161.202.0.1 tunnel 1 remote prefix '161.202.60.0/29'

4. Confirm the establishment of IPsecVPN

At first, you might wanna check IKE(ISAKMP) security associations shown by following.

show  vpn ike sa

Peer ID / IP                            Local ID / IP
------------                            -------------
119.81.00.1                            161.202.00.1                         

    State    Encrypt       Hash    D-H Grp  A-Time  L-Time IKEv
    -----  ------------  --------  -------  ------  ------ ----
    up     aes256        sha1      2        0       3600    1 

Secondly, you can check active IPsec security associations as bellow.

show vpn ipsec sa 

Peer ID / IP                            Local ID / IP
------------                            -------------
119.81.00.1                           161.202.00.1                        

    Tunnel  Id          State  Bytes Out/In   Encrypt       Hash      DH A-Time  L-Time
    ------  ----------  -----  -------------  ------------  --------  -- ------  ------
    1       6           up     0.0/0.0        aes256        sha1      2  21906   1800   

vyatta@sinvyatta1:~$