Stephen Smith's Blog

Musings on Machine Learning…

Archive for October 2017

Getting My Amateur Radio License

leave a comment »

Introduction

I recently passed the exam to get my amateur (ham) radio license. This article is a bit on my experience with taking the course and writing the exam, as well some thoughts on ham radio. People spend a lot of money on cell phones, but most of what they do is done routinely for free by dedicated radio hobbyists. After all a cell phone is just a radio running on frequencies bought up by the cell phone providers.

Why?

There is a perception that amateur radio is an old obsolete dying hobby. That is consists of people using old tube radios trying talk to people far away with lots of interference and poor sound quality.

Why would anyone use ham radio when you can talk to anyone reliably on Skype or FaceTime? Didn’t ham radio only make sense back in the days when long distance calls were prohibitively expensive? Now even most phone plans are quite inexpensive and use the Internet to communicate. So why are the number of amateur radio licenses growing year over year? In the US the number of licenses issued each year is growing at a rate of about 1%, not Internet growth, but pretty steady for something that requires preparing for and writing a written exam.

Here are some of the main reasons for the continuing interest in amateur radio:

  • Disaster preparedness. Most amateur radios can operate off batteries and don’t require cell phone towers, Internet connections or even A/C power.
  • Better radios. There are a proliferation of new radios on the market, at much lower price points (under $100)  and a great many features including computer connectivity, digital channels and more efficient power.
  • Volunteer infrastructure. To use your cell phone you typically pay $75/month to someone for your ability to use their cell phone towers and such. In the amateur radio world many clubs operate repeaters (think amateur cell phone towers) and Internet connections allowing long distance communications with voice or even e-mail from low powered portable radios.
  • Cell phone coverage often isn’t as good as advertised. I do a lot of hiking and mountain biking. Most of the places I go, there is no cell phone coverage. However here on the Sunshine Coast if I get line of sight to the ocean I can probably hit one of the Nanaimo repeaters and communicate. The only other alternative is satellite phones and these make cell phones seem inexpensive.
  • Ability to build your own equipment and experiment. The world of cellphones is very locked down and DIY is not allowed at all. In the world of amateur radio if you have the correct license, you can build your own radio, you can heavily customize your equipment, you can build your own antennas and generally do a lot of experimentation and customization. If you have an interest in understanding how things work and playing with electronics, then this is a great playground.

Licensing

To get my license I took a course offered by VECTOR in Vancouver to prepare and write the basic amateur radio exam. This was a great success and I greatly enjoyed it. VECTOR is an emergency radio non-profit society for the city of Vancouver. The course was taught by a number of VECTOR members usually someone different for each section. This gave a good mix of perspective and provided a good variety over the five morning sessions. The details of this process are a bit different by country so the details here apply specifically to Canada. The basic exam is 100 multiple choice questions, you can take practice exams here. The course covers the main areas of the exam which are: Rules and Policies, Basic Electronics, Antenna Systems, Radio Wave Propagation, Interference and Suppression, Operating Procedures and Station Assembly Practices and Safety. The whole course and exam are theoretical and don’t involve touching a radio whatsoever.

I did pass the exam. If you get 70% then you get access to all frequencies above 30MHz and if you get 80% then you don’t have any frequency limitations. I got over 80% so I can use any amateur frequency band. My call sign is VA6SMI. I have to use this to identify myself in any communications.

To actually build your own radio you need the advanced certification which I might try to get next year. The other qualification you can get is Morse Code. Up till ten years ago this was part of getting a basic license but isn’t required anymore.

Future

I think a lot of the current popularity of ham radio is driven by a wish to be independent of cell phone providers and a wish for much more DIY in our communications. As recent fires, hurricanes and earthquakes have demonstrated the cell phone infrastructure is rather fragile and can take quite some time to get up and running again. During all these emergencies, ham radio was the main form of communications. With global warming, fires and hurricanes will become more common and a need for emergency communications becomes more and more important.

Separately amateurs are adding more and more functionality into ham communications with better digital protocols and speed. They are adding better volunteer infrastructure to support longer distance communications and Internet connectivity. The radio hardware market has gotten much more competitive recently with the addition of several Chinese manufacturers that have driven down prices and driven new innovation.

Will we reach a point where we have “smart” ham radios like we have “smart” phones? Will we be able to run apps like Facebook on our ham radio? Will be be able to use them as ubiquitously as we use our phones? Will we one day be able to break the hold of the cell phone providers and eliminate all those expensive subscription plans? It might take some time but we are slowly moving in that direction. Certainly the ham radio frequencies can’t handle that volume of traffic, but perhaps for those willing to pass the basic radio exam we will start to get some freedom from the current cell phone hegemony.

Summary

I enjoyed the process of getting my ham radio license. I’ve ordered a radio and look forward to getting on the air when it arrives. It was fun reviewing basic electronics and learning more about antennas and radio wave propagation. The rules and regulations part wasn’t bad, once you get the hang of what they are looking for, the questions are quite easy.

Now I’ll see if I’m better connected once I’m out in the woods. Plus I’ll be able to communicate during power outages (we had one today that also took out local cell phone coverage).

 

Advertisements

Written by smist08

October 21, 2017 at 12:47 am

Updates to the TensorFlow API

leave a comment »

Introduction

Last year I published a series of posts on getting up and running on TensorFlow and creating a simple model to make stock market predictions. The series starts here, however the coding articles are here, here and here. We are now a year later and TensorFlow has advanced by quite a few versions (1.3 as of this writing). In this article I’m going to rework that original Python code to use some simpler more powerful APIs from TensorFlow as well as adopt some best practices that weren’t well known last year (at least by me).

This is the same basic model we used last year, which I plan to improve on going forwards. I changed the data set to record the actual stock prices rather than differences. This doesn’t work so well since most of these stocks increase over time and since we go around and around on the training data, it tends to make the predictions quite low. I plan to fix this in a future articles where I handle this time series data correctly. But first I wanted to address a few other things before proceeding.

I’ve placed the updated source code tfstocksdiff13.py on my Google Drive here.

Higher Level API

In the original code to create a layer in our Neural Network, we needed to define the weight and bias Tensors:

layer1_weights = tf.Variable(tf.truncated_normal(
      [NHistData * num_stocks * 2, num_hidden], stddev=0.1))
layer1_biases = tf.Variable(tf.zeros([num_hidden]))

And then define the layer with a complicated mathematical expression:

hidden = tf.tanh(tf.matmul(data, layer1_weights) + layer1_biases)

This code is then repeated with mild variations for every layer in the Neural Network. In the original code this was quite a large block of code.

In TensorFlow 1.3 there is now an API to do this:

hidden = tf.layers.dense(data, num_hidden, activation=tf.nn.elu,
        kernel_initializer=he_init,
        kernel_regularizer=tf.contrib.layers.l1_l2_regularizer(),
        name=name + "model" + "hidden1")

This eliminates a lot of repetitive variable definitions and error prone mathematics.

Also notice the kernel_regularizer=tf.contrib.layers.l1_l2_regularizer() parameter. Previously we had to process the weights ourselves to add regularization penalties to the loss function, now TensorFlow will do this for you, but you still need to extract the values and add them to your loss function.

reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
loss = tf.add_n([tf.nn.l2_loss( tf.subtract(logits, tf_train_labels))] + reg_losses)

You can get at the actual weights and biases if you need them in a similar manner as well.

Better Initialization

Previously we initialized the weights using a truncated normal distribution. Back then the recommendation was to use random values to get the initial weights away from zero. However since 2010 (quite a long time ago) there have been better suggestions and the new tf.layers.dense() API supports these. The original paper was “Understanding the difficulty of training deep feedforward neural networks” by Xavier Glorot and Yoshua Bengio. If you ran the previous example you would have gotten an uninitialized variable on he_init. Here is its definition:

he_init = tf.contrib.layers.variance_scaling_initializer(mode="FAN_AVG")

The idea is that these initializers vary based on the number of inputs and outputs for the neuron layer. There is also tf.contrib.layers.xavier_initializer() and tf.contrib.layers.xavier_initializer_conv2d(). For this example with only two hidden layers it doesn’t matter so much, but if you have a much deeper Neural Network, using these initializers can greatly speed up training and avoid having the gradients either go to zero or explode early on.

Vanishing Gradients and Activation Functions

You might also notice I changed the activation function from tanh to elu. This is due to the problem of vanishing gradients. Since we are using Gradient Descent to train our system then any zero gradients will stop training improvement in that dimension. If you get large values out of the neuron then the gradient of the tanh function will be near zero and this causes training to get stalled. The relu function also has similar problems if the value ever goes negative then the gradient is zero and again training will likely stall and get stuck there. On solution to this is to use the elu function or a “leaky” relu function. Below are the graphs of elu, leaky relu and relu.

Leaky relu has a low sloped linear function for negative values. Elu uses an exponential type function to flatten out a bit to the left of zero so if things go a bit negative they can recover. Although if things go more negative with elu, they will get stuck again. Elu has the advantage that its is rigged to be differentiable at 0 to avoid special cases. Practically speaking both of these activation functions have given very good results in very deep Neural Networks which would otherwise get stuck during training with tanh, sigmoid or relu.

Scaling the Input Data

Neural networks work best if all the data is between zero and one. Previously we didn’t scale our data properly and just did an approximation by dividing by the first value. All that code has been deleted and we now use SciKit Learn’s MinMaxScaler object instead. You fit the data using the training data and then transform any data we process with the result. The code for us is:

# Scale all the training data to the range [0,1].
scaler = MinMaxScaler(copy=False)
scaler.fit(train_dataset)
scaler.transform(train_dataset)
scaler.transform(valid_dataset)
scaler.transform(test_dataset)
scaler.transform(final_row)

The copy=False parameter basically says to do the conversion in place rather than producing a new copy of the data.

SciKit Learn has a lot of useful utility functions that can greatly help with using TensorFlow and well worth looking at even though you aren’t using a SciKit Learn Machine Learning function.

Summary

The field of Neural Networks is evolving rapidly and the best practices keep getting better. TensorFlow is a very dynamic and quickly evolving tool set which can sometimes be a challenge to keep up with.

The main learnings I wanted to share here are:

  • TensorFlow’s high level APIs
  • More sophisticated initialization like He Initialization
  • Avoiding vanishing gradients with elu or leaky ReLU
  • Scaling the input data to between zero and one

These are just a few things of the new things that I could incorporate. In the future I’ll address how to handle time series data in a better manner.

Written by smist08

October 16, 2017 at 9:42 pm