Twitter Follow Button

Monday, 24 June 2013

GW Shore Station MMSI Encoding

In my last post I described how ships using the GW data network encoded their MMSI in GW type 2/101 packets before sending a position report and how Alan W had managed to reverse engineer this. When I wrote that we didn't understand how GW shore stations told ships that they had traffic for them. Clearly the GW network has to be able to tell ships that it has traffic (a telegram or an email perhaps) for them without waiting for an hourly traffic list but how was this done.

 Myself and Alan Ws suspicion had always been that this was that this information was sent in type 5/41 packets sent by shore stations and which looked like this ..

18:18:40 GW Type=5 Count=0 Subtype=41 (10010100101001) (0x9d,0xfa,0xa8,0xb2,0x9b,0xf9)

packets with the same content would be sent for a minute or so then (presumably if there was no reply) a little latter and so. Attempts to decode the MMSI using the GW ship side encoding method resulted in the recovery of an invalid MMSI so it was clear a different encoding method was being used by the shore side.

 The solution to this problem was found once again by Alan W who had the idea that at least some of the time when a ship was told by the shore station that there was traffic for it , that the ship would send back its position and therefore its MMSI. That way we would have the MMSI and would he would be able to work out how it was encoded in the shore side 5/41 packet. Just a couple of days after he proposed using this method Alan W got back to me to say he had worked out how these MMSIs were being encoded.

 Basically its a modification of the ship side encoding method. As before 4 bit nibbles are used to represent numbers in the following way ..

0x0 = "7"
0x1 = "3"
0x2 = "5"
0x3 = "1" or alternate "9"
0x4 = "6"
0x5 = "2" 
0x6 = "4"
0x7 = "0" or alternate "8"
0x8 = "7" 
0x9 = "3"
0xa = "5"
0xb = "1" or alternate "9"
0xc = "6"
0xd = "2"
0xe = "4"
0xf = "0" or alternate "8"

 Notice how this is almost the same encoding map as used for the ship side encoding but the map has been "moved"  along by one digit.

Once again the best way of showing how this is done is with an example. So taking the six byte payload of the 5/41 packet above ..

9d fa a8 b2 9b f9

The hex nibbles are decoded from left to right two digits at a time. As with the ship side encoding the nibbles in each byte are reversed. So the first nibble represents the second number and the second nibble the first number. If the nibble to right of the one being decoded is less than 0x8 then the alternate numbering scheme is used. So from our example 0x9d becomes 23 then 0xfa 50 (so the MMSI so far is 2350) , 0xa8  is 75 (making the MMSI so far 235075) , 0xb2 59 (making the MMSI so far to be 23507559) then we have 0x9b now a MMSI is always 9 digits so has we already have 8 digits we only need to decode 0xb which is 1 making our full MMSI 235075591 which happens to be a vessel called the Kathy C . The final 3 digits which in this case are 330 are some kind of an alert or indicator to the ship telling it what kind of traffic the shore station has for it.

 Now with the latest builds of Rivet if you monitor a GW shore station and have the ships.xml file in the same folder (which can be downloaded from here ) then decoded 5/41 packets will look like this ..

18:18:40 GW Type=5 Count=0 Subtype=41 (10010100101001) (0x9d,0xfa,0xa8,0xb2,0x9b,0xf9)
Traffic For : MMSI 235075591 (KATHY C-2CTI7,Great Britain)
Command=330

Please note that Rivet is a free and totally open source decoder for various HF modes written in Java so it should run on Apple , Linux and Microsoft Windows PCs. The executable JAR file can be downloaded from here  and all the source code is available on Github . If you are interested please join the Rivet mailing list .

Once again many thanks to Alan W for all his work on this.

Tuesday, 14 May 2013

GW MMSI Encoding

In my previous post I discussed the GW protocol used by ships to send position reports and how this was decoded by Rivet the free and open source HF modes decoder. Now the ships identity wasn't encoded into the text string that contains the position report but obviously it had to be in the data exchange somewhere as without it how would GW know which ship was sending the report or which shipping company to pass it on to. My suspicion was that the ships identity was most likely encoded into a data packet called the Type 2 Subtype 101 which was sent prior to each position report. At that time when decoded by Rivet these packets looked like ..

18:10:05 GW Type=2 Count=1 Subtype=101 (10001011100101) (0x85,0x65,0x22,0xac,0x66,0x66)

Note the 6 bytes displayed as hexadecimal numbers within the brackets at the end of the message which is the payload of the packet (i.e the contents). This payload changed with every position report but had similarities so this had to be where the identity was located. Now it seemed to me that the ships identity had to be encoded in one of four ways ..


  1. The ships name
  2. The ships callsign
  3. A unique customer assigned by GW
  4. The ships MMSI.
Now as there are only six bytes in the payload that pretty much rules out the ships name and there weren't enough similarities in the payloads I was seeing for it to be the callsign. Which leaves us with the GW customer number or the MMSI. The most common way of encoding MMSIs into radio data is simply to convert them into 30 bit binary sequences. I took a few 2/101 packets and converted the payload into every possible number from the possible 30 bit sequences in them and used this website to see if any of them were valid MMSI numbers. None were , so that wasn't how they were encoded. Next I wondered if GW were using 4 bit BCD (Binary Coded Decimal) but if that were the case I would only see the numbers 0 to 9 instead I was seeing 0 to F (we are using hexadecimal here) so that wasn't it either.

 At this point I was pretty much stumped and was out of ideas. Thankfully this was when Alan W got in touch with a clever idea. Now just to introduce him Alan W is an ex ships radio officer who has been a co-conspirator of mine on various decoder projects starting with MPT1327 then taxi MDT decoder , DMRDecode and on to Rivet. Alan has an amazing ability to keep working on a problem in a methodical way even when it all seems hopeless. If he was many years older Alan would have been sat in a hut in Bletchley Park working out Enigma keys with pencil and paper but instead he does this. Alan's idea was that since we know the ships position we can use that (via various ship tracking methods) to identify the ship and from that reverse engineer the MMSI from the 2/101 packet. Obviously this doesn't always work as if a ships position report puts it in a giant container port then it could be one of several possible ships but this technique does work often enough to be useful. Now after many many weeks of effort and several phone calls/emails where Alan indicated he was on to something he delivered a bundle of paper work to me. In short Alan by looking at MMSIs and the 2/101 packet payload Alan had found some patterns. To give you a taste of this with this ..

0x00 in the packet represented the figures "33" in the ships MMSI
0x01 in the packet represented the figures "73" in the ships MMSI
0x02 in the packet represented the figures "13" in the ships MMSI

etc etc

In in each 8 byte sequence the 2nd hexadecimal number represents the first of the two MMSI numbers and the first hexadecimal number the second of the MMSI numbers. However he also found some oddities so for instance ..

0x62 in the packet could represent the figures "90" or "10" in the MMSI

From this I was able to put together a little table of the encoding method used ..

0x0 = "3"
0x1 = "7"
0x2 = "1" or alternate value "9"
0x3 = "5"
0x4 = "2"
0x5 = "6"
0x6 = "0" or alternate value "8"
0x7 = "4"
0x8 = "3"
0x9 = "7"
0xa = "1" or alternate value "9"
0xb = "5"
0xc = "2"
0xd = "6"
0xe = "0" or alternate value "8"
0xf = "4"

Which interestingly corresponds to the GW 8 bit alphabet where 0x20 represents "1" , 0x40 represents "2" and so on. Now at this point we could decode any 2/101 packet to a correct MMSI as long as the payload didn't contain a 0x2 , 0x6 , 0xa or 0xe as each could represent one of two numbers. So we had loads of decodes where MMSIs contained the number 1 which should be a 9 and so on. Again Alan was able to use his expertise to would out what was what and so on. Then a couple of days ago I realised what was going on  and how it all worked. Basically when deciding if you need to use the alternate numbers you look a digit ahead (going from left to right) and if the digit is 0x8 or higher you use the alternate number.

The best way of illustrating this is with an example from the 2/101 packet at the start of this page. Its payload is ..

0x85,0x65,0x22,0xac,0x66,0x66

which when made into a single large number is ..

85 65 22 ac 66 66

starting from the left the number 8 represents a 3 and the number 5 a 6 , these need reversing so the MMSI starts "63". Next the 6 represents a 0 (as it is followed by 5 which is less than 8) and the 5 represents a 6. So at this point the MMSI is "6360". Now the 2 represents a 1 (as it is followed by another 2 which is less than 8) but the next 2 is followed by 0xa which is more than 8 so the alternate scheme is used and the number being represented is 9. Now our MMSI is "636091" next we have 0xa which is followed by 0xc so again the alternate scheme is used so we have a "9" and the 0xc represents a "2". Now the MMSI is "63609129". Now MMSIs always contain 9 digits so we have one left to go which is a 0x6 which is followed by a 0x6 so represents a "0". Thus our full MMSI is "636091290" which happens to be a Liberian flagged container ship CMA CGM PARSIFAL.

So there you go that is how it was done and I hope you have enjoyed this. Sorry for rambling on a bit and I hope this reads clearly but it isn't always easy to put a process like this into words.

 Once again many thanks to Alan W for all his help.

 If you want to see the source code for Rivet this can be found here and a pre compiled JAR Java executable file can be download from here.

 For news of Rivet updates and new blog posts please follow me on Twitter.




Friday, 5 April 2013

GW FSK

One of the modes I was asked by Rivet users to support was Globe Wireless (GW) FSK. For those of you who have never heard of them before GW are a commercial company that provide communications services (using HF and satellite) to the shipping industry. You won't be surprised to hear that it is the HF modes that interest me. Now this is where things start to get complicated because GW doesn't just use one HF mode but multiple ones which include ..
  1. 100 baud FSK (200 Hz shift)
  2. 200 baud QPSK
  3. PACTOR
  4. OFDM (22,24,26,28 and 30 carriers)
Now for the time being the mode that interests me is the 100 baud FSK one. This is frequently logged by hobbyists due to the fact that when they aren't sending traffic the shore stations transmit what are often called "Free Channel Markers". This consists of 144 bits (or 18 bytes) + a preamble and takes the form ..

20  38  a3  1f  6c  35  3b  4b  f2  f2  f2  cc  cc  cc  cc  cc  cc  ff

The first 11 bytes as well as the last byte (ff) are always the same and currently their meaning is not understood however the bytes numbered 12 to 17 are the ones indicate the GW shore station sending the transmission. In this case cc indicates the shore station is HEC in Bern , Switzerland. Rivet displays these channel markers in the following way ..

11:34:49 GW Free Channel Marker from Station Code 0xde (SAB, Goeteborg, Sweden)  20  38  a3  1f  6c  35  3b  4b  f2  f2  f2  de  de  de  de  de  de  ff 


These station indicator bytes are interesting though there are 255 possible values sometimes the same ones are used by different shore stations. 

In addition to the 144 bit free channel markers you will also see short 8 bit transmissions (which appear to always be 10010101) which are probably ACKs or acknowledgements that a transmission has been received and also 63 bit + preamble transmissions. It is the latter form of transmission which is the most interesting. These seem to take the form ..

Two bits (always 10) which probably make up the end of the preamble
Twelve bits which appear to determine the type of the packet (i.e what is contained in the payload)
Forty eight bits (or 6 bytes) of actual payload

Of the twelve type packet bits , I found that one bit (no 7) appears to alternate during data transfers and as such is probably a simple counter. One that basis I decided the bits 3 through 6 to represent a number I call the "type" and bits 8 through 14 represent a number I call the subtype. To shorten this I call a packet that is for example type 2 and subtype 101 a 2/101 packet.

The most interesting data sent on the GW FSK networks is probably position reports from ships. To see these you need to find an active GW shore station channel (listen for the free channel markers if you don't hear one within a few seconds that channel is in use) then tune to the corresponding ship channel frequency.  A screen shot of a position report decoded using Rivet can be seen below ..


 When sending such a report a ship initially establishes a connection with a shore station by sending a 5/41 packet. If the shore station hears this then it responds with a 5/41 packet back on the shore side of the link. Now the ship side 5/41 always has the same payload contents which is believed to be "Hello" but the contents of the shore side 5/41 payload change and its purpose is currently unknown. If the shore station does respond then the ship sends a 2/101 packet. We believe this contains the ships MMSI but can't as yet understand how it is encoded (thanks to Alan W for all his ongoing work on understanding this packet).

 Following this the ship sends a succession of 5/86 packets (with alternating count bits) which contain the body of the position report. Finally the ship sends a 2/106 packet possibly to say the report is complete. At this point if everything is OK Rivet will display the entire position report which will look something like ..

<<<<<<$GLOBE,203610.00,A,4927.7702,N,00019.0063,E,000.0,000.0,250113,04.8,W,N*11

What confused me greatly at first was that GW doesn't use ASCII to encode characters but instead uses its own 8 bit character mapping alphabet. With help from Patrick in France we compared the output of a position report decoded in Rivet with one decoded by another program and from that was able to work out much of the encoding system they are using. I won't list the entire alphabet here but if you are interested have a look at the source code for the Rivet GW decoder method getGWChar() which is here.

Another packet you will see is a 9/73 which appears just before another mode (usually OFDM) is used and must be a part of the link negotiation process.

 There is a lot we don't understand about the protocol used by GW FSK. Being able to decode the probable MMSIs in the 2/101 packets would be very useful but we don't understand the contents of the shore side 5/41 or 2/101 packets either.

 Rivet is a free open source decoder for various HF data modes. If you wish to learn more or want to help the project please join the groups mailing list.

Monday, 25 February 2013

An introduction to the Rivet Trigger feature

The recently released build 54 of Rivet (which can be downloaded for free from here) allows the user to add , edit and delete Triggers from within the program. Previously users had to manually edit a XML file using a text editor which I realise isn't easy if you don't have any experience with that sort of thing. Since using Triggers is now within anyone's grasp I thought I would write a little step by step tutorial on how to use it.

 Firstly I had better explain what Triggers are. Rivet has a mode called FSK (Raw) which is intended for users who wish to investigate unknown or little known FSK modes. When selected and after the correct baud rate and shift have been set Rivet displays what the station is transmitting as binary. This is useful when you are initially investigating a mode but soon the amount of data becomes rather overwhelming. To make this mode more useful one of Rivet's users suggested a feature we decided to call a Trigger. Now a Trigger is nothing more than a binary sequence which Rivet stores in its memory and constantly compares with incoming raw FSK data. When the incoming data matches the binary sequence in the trigger then the program does something where the something depends on the type of trigger. So far there are three types of trigger ..


  • Start trigger. With this type of trigger Rivet displays nothing until it receives the sequence of data defined in the start trigger. At which point the program displays the name of the trigger that has been activated and then displays all incoming data.
  • End trigger. When Rivet matches the sequence of data in this type of trigger with the incoming data it displays the name of the trigger and then stops displaying incoming data.
  • Grab trigger. With this type of Trigger when Rivet matches the incoming data with the trigger sequence then it displays the name of the Trigger , the previous backward grab number of bits (a number defined by the user) which were transmitted before the trigger sequence and the forward grab  number of bits (which again has been defined by the user) which follow the trigger sequence. This type of trigger is very useful for packets of data where the synchronisation sequence is in the middle of the packet. 

 So that is what triggers are now let me show you how to use them. For this example we will have a look at a 75 baud 850 Hz shift synchronous FSK signal using KG-84 encryption most likely sent by the armed forces of a NATO member country. These can be found all over the HF bands but if you want to a recording to use with Rivet then one can be downloaded from here.

 To use it start up Rivet then select Raw (FSK) as the decoder mode ..


Next from the Options menu click on the Baudot & FSK Options item. Now select the options "75 Baud" and "850 Hz" as you can see below ..


(Ignore the Stop Bits section which doesn't apply here). Now if you were to load the KG-84 sound sample all you would see is a lot of a binary data. Now it is common knowledge that KG-84 crypto systems use the following binary sequence for synchronisation ..

1111101111001110101100001011100011011010010001001100101010000001

So what we shall do is make this sequence so it is a Rivet start trigger. To do this click on the Triggers menu followed by the "Add , Edit or Delete a Trigger" item. Once you do that then this dialog box will appear ..


Now click on the "Add a new Trigger" button at which point you will see another dialog box which you need to fill in to look like this ..


(you can cut and paste the binary value from this blog post rather than typing it in yourself). After that click on the OK button in this dialog box followed by the OK button in the Modify Triggers dialog box. Once you have done that the program will return to the main screen and automatically save the trigger to your hard disk.

 Next you need to enable the new Trigger. Click on the Triggers menu and you will see the name of your new trigger with a radio button next to it. Click on this to enable the trigger as you see below ..


 With that all done either tune into a KG-84 transmission or load the recording I mentioned earlier. When you do that you will see something like this ..


 You can see from the words "KG-84 Sync" in blue (this triggers name) that the trigger activated several times. The display shows the trigger sequence followed by the data transmitted afterwards. Now that's all well but the display is cluttered with the alternating bits that are transmitted between the messages. It would be better get rid of those. We can do this by setting the alternating sequence of bits ..

1010101010101010

as an end trigger. So when Rivet detects this sequence it stops displaying decoded data until another start trigger sequence is received. To add this trigger click on the Trigger menu then again on the "Add , Edit or Delete a Trigger" item , then on the "Add a New Trigger Button" and fill in the dialog box that appears in the following way ..


Click once again on both OK buttons and then enable the new trigger in the following way ..


Now if you try to decode a KG-84 recording or live transmission you will see something looking like this ..


Both types of triggers can be seen (in blue text) when activated and the KG-84 messages (or blocks of a message) are clearly defined. If you can't be bothered entering these triggers yourself then you can download an existing triggers file from here. Simply put it in the same folder as Rivet and then restart Rivet to use it.

OK that ends this first blog post. I hope to add more posts that act as tutorials on certain Rivet and DMRDecode features as time allows.

Ian