ML: MSDN and Xavier

Microsoft’s MSDN magazine for its development community has been publishing quite a few introductory articles to Machine Learning (ML) over the past few months. January’s issue emphasizes ML with another series of articles, albeit with differing qualities. I liked this quote from the editorial “Advancing AI“: ”ML is a huge graph that requires you to repeatedly examine topics, learning a bit more each time and understanding how topics are interrelated”. An interesting article “Introduction to PyTorch on Windows” is from James McCarthy and only highlights recent rumbles from Microsoft about switching from developing and using its own CNTK library to open source PyTorch, developed mainly by Facebook. In terms of activity, PyTorch statistics on GitHub are about half the activity for TensorFlow and both dwarf CNTK numbers. From this perspective, Microsoft made the right choice abandoning CNTK. For an example of PyTorch staying current, latest version of PyTorch introduces two modes for Python-based ML that enables just in time JIT compilation to improve PyTorch’s adaptability to production environments. Another article “Self-Organizing Maps Using C#” is about an ML technique that is not well known and its usability seems questionable. The third article “Leveraging the Beliefs-Desires-Intentions Agent Architecture” is poorly written and shamelessly plugs the author’s travel agency in the provided sample app.

For most problems in ML, learning is achieved by applying non-linear activation functions to hypotheses. This allows the algorithms to discover non-linearity in data and predict unseen data with greater clarity. Trusted sigmoid activation functions are being replaced by other activation functions and also initialization is being improved. In fact, the “Understanding the difficulty of training deep feedforward neural networks” study details replacing standard random initialization of Deep Learning (DL) networks with other initialization methods. DL networks are neural networks with high number of hidden layers. The study gets rather mathy half way through, but most of the article is digestible. The resulting initialization algorithm is now called Xavier initialization after the first author of the paper and is supported by all leading ML frameworks.

OKRs

Recently I read several articles on OKRs (Objectives and Key Results), a method for setting and tracking the goals leading to better alignment of goals in an organization. OKRs were started by Andy Grove of Intel and first documented in his book “High Output Management” from 1983. The OKRs concept is built on Management By Objectives approach created by Peter Drucker in 50s.

The central principle of OKRs is that each objective (what) is accompanied by several key results (how), that are measurable and can be answered in yes/no format. Great introduction to OKRs is this TED talk by John Doerr, who also wrote a book about it “Measure What Matters”. Doerr has a web site www.whatmatters.com with more content.

OKRs are apparently used by all Google employees. The process of using OKRs at Google is described in the “Startup Lab Workshop” video (over an hour). The video is from 2012 and is considered by some not up to date, but still explains in detail basic concepts.

A web site by one of the proponents of OKRs who makes living teaching companies how to use them is here. It has many interesting articles, including an article about using Shared OKRs from an ex-employee of Google.

2018-01-14: Updated category.

Really Old Geek

The New York Times published an article about Donald Knuth “The Yoda of Silicon Valley”. The article might idolize his work and impact, but provides a great overview of his work. Mr. Knuth is a living legend in the field of computer science, known mainly for his seminal book on algorithms “The Art Of Computer Programming (TAOCP)”.

Given Knuth’s life-time work on algorithms, the following quote from him might be surprising: “I am worried that algorithms are getting too prominent in the world. It started out that computer scientists were worried nobody was listening to us. Now I’m worried that too many people are listening.” I think it is in direct reference to usage of algorithms and their biases in Machine Learning (ML).

For a different take on TAOCP, read this post on Medium, where the author jokes that a lot of people claim how important the book is, without actually reading it.

Why is smart pointer better than a raw one?

In a previous post I suggested that raw pointers should be replaced by smart pointers. This post explains why smart pointers are better than raw ones and how to convert an existing code using raw pointers to use smart pointers. For the sake of this post, when I refer to smart pointers, I talk about unique pointers, unique_ptr.

Why is smart pointer better?

Smart pointers have one crucial advantage over raw pointers: they guarantee that their destructors are called during stack unwinding. That means that any memory allocated in constructors will be automatically freed.

Let’s look at sample code, that illustrates the advantage of smart pointers over raw pointers. First, let’s define a simple class that owns some memory, say std::string to store text.

class Test
{
 string s;
public:
 Test(string sVariable) :
  s(sVariable)
 {
  cout << "Constructor (" << s << ")..." << endl;
 };
 ~Test()
 {
  cout << "Destructor (" << s << ")..." << endl;
 };
};

Now assume that the main function will call myFunction() method to perform some operations. Main will catch any exceptions generated in myFunction() and exit the program.

int main()
{
 try
 {
  myFunction();
 }
 catch(...)
 {
  cout << "Exception handled" << endl;
 }
 
 cout << "End" << endl;
 return 0;
}

Now let’s declare two pointers to Test class in myFunction(). First is rawPointer, which is raw pointer, and then smartPointer which is a smart pointer. Then let’s throw an exception to simulate some unexpected behavior of myFunction. At the end of the function we delete rawPointer. Note, that a smartPointer variable is not deleted as its destructor is guaranteed when the variable goes out of scope.

int myFunction()
{
 cout << "raw pointer" << endl;
 Test *rawPointer = new Test("raw pointer");
 
 cout << "smart pointer" << endl;
 unique_ptr<Test> smartPointer = make_unique<Test>("smart pointer");
 
 cout << "Throw!" << endl;
 throw;
 
 cout << "Delete raw pointer" << endl;
 delete rawPointer;
 
 return 0;
}

When we run this code, we see the following output:

raw pointer
Constructor (raw pointer)...
smart pointer
Constructor (smart pointer)...
Throw!
Destructor (smart pointer)...
Exception handled
End

Notice, that rawPointer variable is never deleted, causing a memory leak to occur.

How to convert raw pointers to smart pointers?

The above code provides a template how to convert existing code that is using raw pointers to smart pointers. To simplify the instructions, let’s use an example of code using raw pointers to int.

  • Convert raw pointer declaration and allocation from
    int * rP;
    rP = new int;

    to
    unique_ptr<int> sP = make_unique<int>();
  • Remove delete call to release rP memory

That’s it! The code after the conversion is simpler and more robust, as you will never forget to delete the memory, or delete it twice, or the delete will simply be skipped by unexpected code flow (see the example using throw).

When is C pointer valid?

C is largely based on raw pointers and one of the most often asked questions is when is a pointer a valid one. The reason is that dereferencing an invalid pointer generates an access violation exception (0xC0000005) and if not caught, the program crashes.

You quickly learn there is IsBadReadPtr function available and its variants. You look at the name and think you found the best way to check the pointer’s validity. That is until you notice, that Microsoft documentation says these functions are obsolete and warns against using them in no uncertain terms. And Raymond Chan from Microsoft provides details at his blog The Old New Thing.

Then, you think this is such basic C/C++ question, there must be a way to do it. You start searching the Internet and slowly come to realize, that there is no way to check validity of raw pointers.

However, there are best practices how to guard against invalid pointers.

Best practices to guard against bad pointers

  • Initialize all pointers to nullptr.
  • After deleting an object, set its pointer to nullptr.
  • Check pointer for non-zero value before dereferencing it.

The above best practices will not prevent you from dereferencing invalid raw pointers altogether. For example, it does not prevent somebody from passing an invalid pointer through the API. In that situation, you have to accept GIGO, garbage in garbage out paradigm.

How to check if pointer has non-zero value?

There are few ways of checking if a pointer points to nothing. If it does not, it has a non-zero value. Still does not make it valid, but you have done your best.

  • if (p == 0): Check against number 0 which is typed as int. OK approach.
  • if (p == NULL): NULL is macro defined as 0. Slightly better, makes your application more readable.
  • if (p == nullptr): Check against C++11 keyword. Best, clean approach.

The above triad is nicely explained by Kate Gregory in her “C++ Fundamentals” class on Pluralsight, in chapter “Pointers”. For another take on the same read Bjarne Stroustrup.

Conclusion

There is no good way to check if a raw pointer is valid or not, in other words if it can be safely dereferenced or not. You should seriously consider not using raw pointers in your code and start using smart pointers unique_ptr and shared_ptr from C++11.

Hello programmers!

Welcome to CodingRestart.com, blog for seasoned developers that want to upgrade their skills.

If any of the following is true for you, this site is for you:

  • You feel threatened by younger colleagues, because you do not know popular programming languages and buzzwords they use.
  • You think Kernighan and Ritchie’s “The C Programming Language” is the latest book on C/C++.
  • When writing new code, you frequently copy and paste snippets from the old code, preferably yours.
  • You swear this is not how you loop through characters in a string in C++:
    for (auto my_char : my_string)
  • When you hear lambda, you think Greek life.

I started on this journey more than a year ago and it has been a great experience. I have significantly upgraded my coding skills and increased my enjoyment at work. I believe with the right mindset and few hours of learning each week, you can do the same.

Hope you will check back soon!