Stephen Smith's Blog

Musings on Machine Learning…

Open Source Photography Toolkit

leave a comment »

Introduction

Since retiring, I’ve switched to entirely running open source software. For photography, Adobe Photoshop and Lightroom dominate the scene. Most articles and books are based on these products. The Adobe products have a reputation for being very good, but they are quite expensive, especially since they have switched to a subscription model of pricing. In this article I’m going to talk about the excellent open source programs that work very well in this space.

Basically there are two streams here, the quicker and easier software equivalent to Adobe Lightroom and then the more technical and sophisticated software equivalent to Adobe Photoshop.

I run all these programs on Ubuntu Linux, however they all have versions for the Mac and Windows.

You can download the source code for any open source program and have a look at how the programs work. If you find a bug, you can report it, or if you are a programmer you can fix it. Figuring out enough of a program to work on it is a large undertaking, but I feel comforted that that avenue is open to me if I need it.

digiKam

digiKam is an open source photo management program similar to Adobe’s Lightroom. It is easier to use than a full photo editing tool like GIMP or Adobe Photoshop, and has tools to automate the processing of the large number of photos taken in a typical shoot. It has the ability to import all the photos from raw format for further processing, it has a pretty good image editor built in and then lots of tools for managing your photos, like putting them in albums, assigning keywords, and editing the meta-data. There is an extensive search tool, so you can find your photos again if you forgot where you put them. There are tools to publish your photos to various photography websites as well as various social media websites.

screenshot from 2019-01-05 11-57-36

Unlike Lightroom, there aren’t nearly as many books or tutorials on the product. I only see one book on Amazon. However the web based manual for digiKam is pretty good and I find it more than enough. It does peter out near the end, but most of the things that are TBD are also easy to figure out (mostly missing the specifics of various integrations with third party web sites).

Another difference is that digiKam does actually edit your pictures and doesn’t just store differences like LR does, so you need to be aware of that in your management workflows.

Lightroom costs $9.99/month and is subscription based. digiKam is free. One benefit is you don’t have to worry about having your photos held hostage if you get tired of paying month after month. Especially if you are an infrequent user.

GIMP

GIMP is very powerful photo-editing software. It is an open source equivalent of Adobe Photoshop. I recently saw a presentation by an author of a book on Photoshop on his workflow for editing photos with Photoshop. I was able to go home and perform the exact same workflows in GIMP without any problems. These involved a lot of use of layers and masks, both of which are well supported in GIMP.

screenshot from 2019-01-05 12-10-31

Both Photoshop and GIMP are criticised for being hard to use, but they are the real power tools for photo editing and are both well worth the learning curve to become proficient. There are actually quite a few good books on GIMP as well as many YouTube tutorials on the basic editing tasks.

For 90% of your needs, you can probably use digiKam or Lightroom. But for the really difficult editing jobs you need a tool like this.

Photoshop typically costs $20/month on a subscription basis. GIMP is free.

RawTherapee

GIMP doesn’t have the ability built in to read raw image files. There are plug-ins hat you can install, but I’ve not gotten good results with these, often they work stand-alone, but not from within GIMP. digiKam can process raw files, and doing that en-mass is one of its main features.

screenshot from 2019-01-05 14-02-19

Sometimes you want a lot of control of the process when you do this processing. This is where RawTherapee comes in. It is a very sophisticated conversion program. It supports batch processing and has very sophisticated color processing.

Often in the open source world, components are broken out separately rather than bundled into one giant program. This provides more flexibility to mix and match software and allows the development teams to concentrate on what they are really good at.

Typically you would take all your pictures in your camera’s raw mode, convert these to a lossless file format like TIFF and then do your photo editing in GIMP. This is the harder, but more powerful route as opposed to using digiKam for the entire workflow.

OpenShot

OpenShot is actually movie editing software. I included it here, because many photographers like to create slideshows of their work, where the images have nice transitions and change from image to image with the music. OpenShot is an ideal open source program for doing this. If you have a Mac, then you can use iMovie for this, but if you don’t have a Mac or what something that works on any computer then OpenShot is a good choice.

screenshot from 2019-01-05 14-08-30

Summary

There are good open source pieces of software that are very competitive with the expense commercial software products. Adobe has a near monopoly in the commercial space and tries to squeeze every dime it can out of you. It’s nice that there is a complete suite of alternatives. I only use open source software for my photography, and have find it to easily fill all my needs.

This article only talks about four pieces of software. There are actually many more specialized applications out there that you can easily find by googling. Chances are if you look below the ads in your Google search results, you will find some good free open source software that will do the job for you.

 

Advertisements

Written by smist08

January 5, 2019 at 10:29 pm

The Technology of “Influence” – Part 4 Kali Linux

with one comment

Introduction

In my novel “Influence”, the lead character J@ck Tr@de performs various hacking tasks. In the book he spends a lot of time securing his connections, hiding his identity and hiding his location. In this series of blog posts, I’m going to talk about the various technologies mentioned in the book like VPN, the Onion Browser, Kali Linux and using VHF radios. I’ve talked about HTTPS,  VPNs and the Onion Browser so far, now we’re going to discuss Kali Linux.

Linux is an operating system like Windows or MacOS. An operating system manages the hardware on your computer and manages running the applications that you use like a word processor or Internet browser. Linux is open source and free. There are many distributions of Linux, that are complete pre-built systems for you to install. The differences between the different distributions include things like how the desktop is configured to look, which other open source programs are bundled, when updates are installed, how updates are installed and how the system is configured. Kali Linux is one of these distributions that emphasizes security and comes with all the common open source security and hacking tools pre-installed.

Most hackers consider Linux better suited to their needs than Windows or MacOS. They don’t trust Microsoft or Apple to do a good enough job with security or worry about these big corporations spying on them. With Linux it’s easy to do things like change your MAC address and run the tools to keep you safe, secure and anonymous.

I blogged about Kali Linux for the Raspberry Pi last year here. J@ck would use this on the Raspberry Pi’s he has the homeless people plant in the garbage near coffee shops to tap into their wifi.

For anyone interested, my book is available either as a paperback or as a Kindle download on Amazon.com:

Paperback – https://www.amazon.com/dp/1730927661
Kindle – https://www.amazon.com/dp/B07L477CF6

 

Offensive Security

The philosophy behind Kali Linux is that for your network to be secure, you have to attack it like a hacker. You have to use all the tools in a hacker’s toolbox, to ensure hackers can’t break in. Setting up security isn’t just a matter of following a checklist of todo items. You have to think like a hacker and try to penetrate your security like a hacker. Or hire so called white hat hackers to do it for you. Generally it’s a good practice to get a second or third pair of eyes looking for holes and weaknesses. The good white hat hackers are in high demand, and don’t come cheap.

Kali Linux comes with all the common open source hacking tools pre-installed. So they are all there and ready to attack your network. Of course the advertising is all about white hat hackers using these for good. But, of course, this is the same Linux distribution and toolset used by most of the malicious black hat hackers.

Kali Linux is also fairly secure if you follow the various instructions during installation, about securing things with private/public keys and such. Kali Linux doesn’t install any application servers like web servers or database servers, since these are usually good targets for hackers to attack.

Kali Linux is based on Debian Linux, so you can do most of the things other Debian based distributions can do, like Ubuntu. Just without all the useful productivity applications pre-installed. Kali Linux has versions for small system on a chip (SoC) like the Raspberry Pi. In these versions, any tools that won’t run well on the more minimal hardware are left out.

Thinking Like a Hacker

You can find quite a few books on how to use all the tools installed with Kali Linux. These are all a good start, but like I said, setting up a recipe or checklist is insufficient. You have to learn to think like a hacker. You have to figure out how to find the weak points in a network and then how to keep poking at them from all sorts of angles until you can penetrate them. Remember the world of hacking isn’t static. Hackers are always discovering new techniques and new weaknesses to exploit. If you are serious about protecting your network’s security then you have to stay on top of the latest developments. Often the weak points aren’t in the software, but in the employees. Hackers will use so called social engineering attacks to trick you users into revealing their passwords or other key information. Perhaps the hacker will leave a few USB keys lying around, that contain viruses that will infect your network if plugged into a corporate computer. Perhaps the weakness is a third party piece of hardware like a network router or firewall. These are notorious for having backdoors or other security weaknesses. You have to ensure all these miscellaneous pieces of equipment are kept up to date, or replaced if a serious problem is discovered.

The Security Onion

A key metaphor in the security industry is that you want to design your security systems like an onion with multiple layers, and not like and egg with one shell, which once breached gives access to everything inside.

Perhaps at the outside of your network, there are secure firewalls, but then inside that there are products that detect malicious or suspect network traffic and set off alerts when discovered. Further all the servers on the networks have very few ports open for network traffic and all the ones that are open are configured to use quite strong forms of authentication. Its common to use two level authentication, where the user needs a code from their cell phone in addition to their password in order to logon. Perhaps the parts of the network aren’t connected, so if an intruder gets access to one server, he’s still isolated from all the others.

Designing secure systems is an art as well as a science. The good news is that there are many open source tools available to set up all these layers of security. So it doesn’t have to be expensive, except where you have to hire the people to put it all in place.

Summary

Kali Linux is the preferred Linux Distribution of hackers. It pre-installs all the common open source hacking tools and by default has a fairly secure configuration. Of course any hacker will further secure their system and install a few more specialty tools perhaps from the dark web or things they wrote themselves.

Written by smist08

January 2, 2019 at 11:47 pm

The Technology of “Influence” – Part 3 The Onion Browser

with 2 comments

Introduction

In my novel “Influence”, the lead character J@ck Tr@de performs various hacking tasks. In the book he spends a lot of time securing his connections, hiding his identity and hiding his location. In this series of blog posts, I’m going to talk about the various technologies mentioned in the book like VPN, the Onion Browser, Kali Linux and using VHF radios. I’ve talked about HTTPS and VPNs so far, now we’re going to discuss the Onion Browser and the Tor network.

For anyone interested, my book is available either as a paperback or as a Kindle download on Amazon.com:

Paperback – https://www.amazon.com/dp/1730927661
Kindle – https://www.amazon.com/dp/B07L477CF6

The Tor Network

Tor is an abbreviation for The Onion Router. You tend to see Tor and Onion used interchangeably. Nowadays Tor tends to refer to the Tor network and Onion to the open source browser that utilizes the Tor network to browse the web.

The Tor network and Onion Browser were developed by a group of people dedicated to security, privacy and anonymity. The Tor network depends on thousands of volunteers operating Tor network nodes (servers). When you use the Onion browser, each server connection that you use goes through a different random path through these Tor network nodes. Each node acts like a VPN, encrypting communications and hiding the location of the original request. To some degree using the Tor network is like using a set of different VPNs for each website you visit. This makes tracking you down very hard.

The Onion Browser is an open source Internet browser that performs all it’s requests through the Tor network.

The Dark Web

The dark web consist of a number of websites that aren’t linked to from the regular web. They only accept requests over the Tor network and you have to find out about them through means other than Googling. This so-called dark web has been know to host all sorts of “bad” e-commerce sites dealing in illegal drugs, human trafficking and child pornography. Whenever law enforcement tries to ban encryption or anonymity, they always use these sites as excuses to be able to track and spy on normal people’s web activity.

On the other hand in highly repressive states which block a lot of Internet traffic with the outside world, the Tor network and the dark web are the only way that dissidents can freely communicate, or that regular citizens can browse the web at all. Generally governments spend way more time tracking dissidents than they ever spend tracking down the illegal websites they claimed to be upset about.

How Safe Is It?

That all sounds pretty good, so why doesn’t J@ck just use the Onion browser and just not bother with all the other things he does? For one thing, government security services spend a lot of time trying to crack the Tor network. Many of the thousands of nodes in the Tor network are actually operated by government agencies. If one of these is your exit node, then they can get quite a bit of info on you. It’s a bit of a race between the developers of the Tor network and government departments like Homeland Security as to how safe the network is at any time.

Another problem is that even though, say Google can trace who you are from the network traffic, they can record things like your typing patterns and mouse movement patterns. These are apparently just like fingerprints and can be used to identify you. Other means are required to disguise these sort of things.

A general maxim in security is never trust anything entirely. The original name of the Onion browser was based on this idea of having many layers of security like the layers of an Onion. Tor provides several layers, but you can add more layers to be more secure.

Performance

Every server that you hit introduces a delay as that server receives, processes and then transmits your network packets of information. With the Tor network, you introduce a bunch of these delays to give you better security and privacy. Further, not all the Tor nodes have the greatest Internet bandwidth or server power. After all they are paid for and operated by volunteers. This all adds up to the Tor network being very slow. If you ever try to download a movie of the Tor network it will take forever. This is why people pay for VPNs with decent bandwidth and performance, rather than using Tor. If you aren’t downloading movies, and just doing small queries then it is usable. This is what J@ck tends to be doing.

Summary

The Tor network and Onion Browser are key tools used by every hacker. It provides great security and anonymity at the cost of access speed. If you want to check out the dark web then you need to use the Onion Browser.

Written by smist08

December 22, 2018 at 2:57 am

The Technology of “Influence” – Part 2 VPN

with 3 comments

Introduction

In my novel “Influence”, the lead character J@ck Tr@de performs various hacking tasks. In the book he spends a lot of time securing his connections, hiding his identity and hiding his location. In a series of blog posts, I’m going to talk about the various technologies mentioned in the book like VPN, the Onion Browser, Kali Linux and using VHF radios. I talked about HTTPS in my last post and in this article, we’re going to discuss Virtual Private Networks (VPNs).

For anyone interested, my book is available either as a paperback or as a Kindle download on Amazon.com:

Paperback – https://www.amazon.com/dp/1730927661
Kindle – https://www.amazon.com/dp/B07L477CF6

What is a VPN?

We talked about HTTPS last time as a way to secure the communications protocol that a Browser uses to talk to a Web Server. Now consider a corporate network. People at work have their computers hooked directly into the corporate network. They use this to access email, various internal corporate websites, shared network drives and other centrally deployed applications. All of these services have their own network protocols all different than HTTP. Some of these protocols have secure variants, some don’t. Some have heavy security, some light security. Now suppose you want to access these from home or from a hotel while on a business trip? You certainly can’t just do this over the Internet, because its a public network and anyone can see what you are doing. You need a way to secure all these protocols. This is the job of VPN. When you activate VPN on your laptop, it creates a secure tunnel from your laptop through the Internet to a server in your secure corporate data center. The security mechanisms VPN uses are largely the same as HTTPS and pretty secure. Using VPN then allows you to work securely from home or from remote locations while travelling.

Why Would J@ck Use VPN?

J@ck Tr@de doesn’t work for a corporation. Why does he use VPN? Whose VPN does he use? In the example above, if I’m connected to my corporate VPN, all my network traffic is tunnelled through the VPN to the corporate server. So if I browse the Internet while connected to VPN, my HTTPS requests are sent to the corporate server and then it sends them to the Internet. This extra step slows things down, but it has an interesting side-effect. If I’m not signed into Google and I Google something, Google will see my Internet Address as the corporate server rather than my laptop. That means Google won’t know who I am exactly. It also means my location shows up as the location of the corporate server. This then hides both my location and my identity, things J@ck is very interested in doing.

But J@ck doesn’t work for a corporation? Whose VPN does he use? This “feature” of hiding identity and location is sufficiently valuable that people like J@ck will pay for it. This has resulted in companies setting up VPNs just for this purpose. Their VPN server doesn’t connect to other corporate network programs, only the Internet. Using one of these VPN services will help hide your identity and location, or at least websites can’t determine these from the address fields in your web network packets.

VPNs are popular with non-hackers as well to get at geographically locked content. For instance if you live in Canada, then the content you can get from Netflix is different than the content you get in the USA. But if you are in Canada and connect to a US based VPN server then Netflix will see you as being located in the USA and will give you the US content while you are connected.

Downsides of VPN

Sounds good, so what’s the catch? One is that since these are usually paid services, so you need to pay a monthly fee. Further, you need to authenticate to the VPN service so they know who you are. The VPN knows your IP address so it can trace who and where you are.

So do you trust your VPN? Here you have to be careful. If the VPN provider is located in the USA, then its subject to the Patriot Act and law enforcement can get ahold of their info. If you want US Netflix content, then you have to use an US based VPN, but at the same time US law enforcement really doesn’t care that much about the vagaries of what Netflix allows where. If you are a hacker then you really care and probably want to use a VPN in a country with some protections. For instance in Europe, getting a warrant for this is very difficult. Or perhaps use a VPN in the Caribbean that tend to ignore external law enforcement agencies requests. A bit of Googling can help here. Some hackers use a two or three VPNs at once, located in wildly different jurisdictions to make it even harder to be traced.

Internet bandwidth is expensive, so feeding streaming movies through a VPN can require their delux expensive plan. Doing little bits of hacking doesn’t require that much bandwidth so can be a little cheaper.

There are free VPNs, but most of these are considered rather suspect since they must be supporting themselves somehow, perhaps by selling secrets. VPNs are illegal in some countries like Iraq or North Korea. VPNs are required to be run by the government in other countries like China and Russia. So be wary of these.

Summary

VPNs are a way to secure your general Internet communications. They have the desirable side-effect of hiding your Internet address and location. VPNs are absolutely necessary for corporate security and useful enough that lots of other people use them as well,

Notice that J@ck doesn’t just rely on an VPN by itself, rather its one layer in a series of protections to ensure his anonymity and privacy.

Written by smist08

December 13, 2018 at 12:34 am

The Technology of “Influence” – Part 1 HTTPS

with 5 comments

Introduction

In my novel “Influence”, the lead character J@ck Tr@de performs various hacking tasks. In the book he spends a lot of time securing his connections, hiding his identity and hiding his location. In a series of blog posts, I’m going to talk about the various technologies mentioned in the book like VPN, the Onion Browser, Kali Linux and using VHF radios. But first I need to talk about HTTPS which is the normal Internet security mechanism we all use to secure our bank and shopping transactions. I’ll look at what this does protect and what it doesn’t protect. Once we understand the limitations of HTTPS, we can go on to look at why J@ck goes to so much trouble to add so many extra levels of security and misdirection.

For anyone interested, my book is available either as a paperback or as a Kindle download on Amazon.com:

Paperback – https://www.amazon.com/dp/1730927661
Kindle – https://www.amazon.com/dp/B07L477CF6

What is HTTPS?

The communications protocol that Browsers use to communicate with Web servers is called HTTP (HyperText Transfer Protocol). This is the protocol that gets data for websites and downloads it to your browser to be displayed. The S added is for Secure and makes this process secure by encrypting the communications. In the early days of the Web doing all this encrypting/decrypting was expensive both for typical personal computers of the day and for web sites that have quite a high volume of traffic. These days computers are more powerful and can handle this encryption easily, and due to the prevalence of hackers and scammers, the current tendency is to just encrypt all Internet traffic. In fact most modern browsers will not let you use plain old HTTP and require the S for security.

HTTPS is actually quite secure. It is very difficult to decrypt with modern computer resources (even cloud based). It authenticates the server via a digital certificate which is provided by a certificate authority that validates the identity of who has the certificate. The protocol protects against man-in-the-middle attacks where someone impersonates one party and relays the information. It protects against data being tampered with in any way.

Sounds pretty good, and in fact it is pretty good. So why does J@ck feel a need to use VPNs or use the Tor network via the Onion Browser?

Weaknesses of HTTPS

J@ck’s main complaint is that: who he talks to knows who he is and what he is doing. For instance, all Google searches go through HTTPS, so no one can eavesdrop on what you are searching for. But, Google knows. Google logs all your searches and builds a detailed profile of you. Further Google is an American company and subject to the Patriot Act and other government programs to hand over your data if requested. Hence if, say you are Googling on hacking techniques, Google could turn that over to the FBI along with your IP address. Then the FBI can ask your ISP who owns this IP address and identify you and come to your door to ask you some questions. Of course if you are signed into your Google account, then they don’t need to bother with the IP address lookup. J@ck certainly doesn’t want that to happen.

HTTPS has some other weaknesses as well. The process of granting authentication certificates isn’t perfect. One of the most common Windows Updates is to alter the list of trusted certificate authorities, since they are often caught handing out fake certificates to shady operators. Along the same lines, most people don’t check the certificate of who they are talking to. This is how most phishing emails work. They send and email asking you to check your bank account, with a link that is similar to your banks, but not the same. The fake link goes to a page that looks like your bank’s login page, but it isn’t. If you click on the certificate icon in your browser you will see the certificate that that it isn’t your banks. But who does this? If you type in your username and password to this site, the bad actors can then use it to login to your real bank account and steal your money.

Hackers can learn a bit about the content of HTTPS traffic even though its encrypted. Perhaps the URI by comparing the lengths of the strings.

Another worry is that often more companies can see your data than you might think. For instance if you are talking to your bank, then you certainly expect you bank can understand your data. However your bank might use a third party web hosting company to host the web site and then that company can also see your data. Then the web hosting company might host the site on a cloud provider like AWS or Azure and then that group might be able to see your data. Then often websites protect themselves against DDoS attacks using a service like CloudFlare and part of that setup lets CloudFlare see the unencrypted data. So suddenly you aren’t just trusting one company, but four companies. This then provide many more vectors of attack and vulnerable points for hackers. Plus the bank might have hired outsourced programming to set up their website, and those contractors have enough credentials to see unencrypted data. These are actually the main causes for all the security breaches you read about at large Internet sites.

Summary

HTTPS is a pretty good way to secure Internet traffic and if you follow some basic good practices you should be ok. For instance never use a link in an email. Always goto the website through another means (like a favorite or use Google). For data you really care about, like your bank account, only access it from a network you trust, not the Wifi at a hotel or coffee shop.

Now that we understand the strength and weaknesses of HTTPS we can look at the extra layers that J@ck uses to stay anonymous and secure.

Written by smist08

December 11, 2018 at 2:33 am

Posted in Security, Writing

Tagged with , ,

“Influence” – My First Novel

with 10 comments

Introduction

I am really excited that my first novel is now available for sale on Amazon as either a paperback or as a Kindle download. Here is the synopsis for the novel:

Influence is set in the present about punk rock hacker, J@ck Tr@de, who discovers a security backdoor in a large corporate server operating system to gain access to all of the world’s servers. He uses this illicit access to mine bitcoin and influence local politics via Social Media. He becomes criminally and romantically entwined with Mia, the creator of the backdoor and their plans escalate to increase their wealth and power. The FBI investigate and chase them, in a clumsy cat and mouse game. As the story progresses, J@ck’s Social Media altering Bots become more and more influential.l. They make J@ck a billionaire through stock market manipulation. The Social Media Bots continue to evolve…

Where It Began

I’ve been writing this blog since January, 2009. That will be ten years of blogging next month! I really enjoy blogging, mostly on my technology interests. This blog started by being all things ACCPAC, since that’s what I worked on originally at Computer Associates, then ACCPAC International and finally Sage. I find I really enjoy writing and was looking to do more. Almost three years ago I retired, and at that point mostly lost my main blogging topic on Sage 300 (ACCPAC).

I’ve always been a big Science Fiction fan, I’ve read Science Fiction since elementary school with books like Isaac Asimov’s Lucky Starr series. When I started at Computer Associates, I lived in Tsawwassen and had a long bus ride commute downtown. I spent most of this ride voraciously reading all the Science Fiction novels nominated for the Hugo and Nebula Awards, as well as whatever my favorite authors published.

We spent this March down in Yuma, Arizona. One of the things we did while we were down there was attend the “Write On the Edge” writing group. This group gets together weekly at the Yuma Foothills Library to do some writing. They do some sort of writing exercise each meeting. The first time we attended, it was to write a few paragraphs on a topic that a moderator chose. Since Easter was approaching, the topic was “Easter Eggs”. There were a lot of short pieces on people’s favorite Easter family moments (whether real or imagined) and one about horrible carnivorous beasts hatching from the eggs. I took the approach of computer software Easter Eggs meaning little jewels buried in the code. This led to the creation of the J@ck Tr@de character and the few paragraphs around where he finds the W-Server backdoor.

Then as we did the 24 hour drive back home (over three days), I kept thinking about those few paragraphs and felt I had enough ideas to develop it into a novel. This then led to “Influence”.

A Lot of Writing

When I got home, I put the blogging aside (hence no articles here from March to July) and started my novel. I first wrote a very quick outline. Mostly a beginning and an ending, some settings and some notes on some characters. I then started writing. I tried to write at least two pages a day. Sometimes more, and there were only a couple of days when I didn’t write anything. I participated in a couple of writing groups, one in Gibsons and one in North Vancouver. These were get togethers at coffee shops where writers bring their laptops and write. My wife, Cathalynn Labonte-Smith is also an author. She has Creative Writing, Technical Writing and Teaching degrees. She has worked as an editor and would read what I wrote each day and edit it. I wrote the whole thing in Google Docs, so the collaboration was really easy. I was pretty happy to finish my first and second drafts in July.

It’s Written, Now What?

I then went to publish the book. Most of the bigger publishers only take submissions from literary agents. So I followed the submission guidelines for a number of agencies, but didn’t have any luck. I did quite a bit of online research and talked to quite a few authors. The consensus seemed to be that the publishers were publishing less and less each year and that they only picked up authors who’ve already made a name for themselves on their own. Further the published authors didn’t think the publishing companies do much to promote their work and that they have to do all their own promotion. Meanwhile self-publishing is getting easier and easier. I chose Kindle Direct Publishing form Amazon, mostly because its all online, there are no upfront costs and it was easy. So now my book is on Amazon for sale around the world.

Summary

Buy my book:

Paperback – https://www.amazon.com/dp/1730927661

Kindle – https://www.amazon.com/dp/B07L477CF6

It was a lot of fun writing. I planned this book to be the first book in a trilogy and I’ve already written sixty pages of the second volume.

 

Written by smist08

December 5, 2018 at 7:08 pm

Getting Productive with Julia

with 3 comments

Introduction

Julia is a programming language that is used quite extensively by the scientific community. It is open source, it just reached its version 1.0 milestone after quite a few years of development and it is nearly as fast as C but with many features associated with interpretive languages like R or Python.

There don’t seem to be many articles on getting all up and running with Julia, so I thought I’d write about some things that I found useful. This is all based on playing with Julia on my laptop running Ubuntu Linux.

Run in the Cloud

One option is to avoid any installation hassles is to just run in the cloud. You can do this with JuliaBox. JuliaBox gives you a Jupyter Notebook interface where you can either play with the various tutorials or do your own programming. Just beware the resources you get for free are quite limited and JuliaBox makes its money by charging you for additional time and computing power.

Sadly at this point, there aren’t very many options for running Julia in the cloud since the big AI clouds seem to only offer Python and R. I’m hoping that Google’s Kaggle will add it as an option, since the better performance will open up some intriguing possibilities in their competitions.

JuliaBox gives you easy direct access to all the tutorials offered from Julia’s learning site. Running through the YouTube videos and playing with these notebooks is a great way to get up to speed with Julia.

Installing Julia

Julia’s website documents how to install Julia onto various operating systems. Generally the Julia installation is just copying files to the right places and adding the Julia executable to the PATH. On Ubuntu you can search for Julia in the Ubuntu Software App and install it from there. Either way this is usually pretty easy straight forward. This gives you the ability to run Julia programs by typing “julia sourefile.jl” at a command prompt. If you just type “julia” you get the REPL environment for entering commands.

You can do quite a lot in REPL, but I don’t find it very useful myself except for doing things like package management.

If you like programming by coding in your favorite text editor and then just running the program, then this is all you need. For many purposes this works out great.

The Juno IDE

If you enjoy working in full IDE’s then there is Juno which is based on the open source Atom IDE. There are commercial variants of this IDE with full support, but I find the free version works just fine.

To install Juno you follow these instructions. Basically this involves installing the Atom IDE by downloading and running a .deb installation package. Then from within Atom, adding Julia support to the IDE.

Atom has integration with Julia’s debugger Gallium as well as provides a plot plane and access to watch variables. The editor is fairly good with syntax highlighting. Generally not a bad way to develop in Julia.

Jupyter

JuliaBox mentioned above uses Jupyter and runs it in the cloud. However, you can install it locally as well. Jupyter is a very popular and powerful notebook metaphor for developing programs where you see  the results of each incremental bit of code as you write it. It is really good at displaying all sorts of fancy formats like graphs. It is web based and will run a local web server that will use the local Julia installation to run. If you develop in Python or R, then you’ve probably already played with Jupyter.

To install it you first have to install Jupyter. The best way to do this is to use “sudo apt install jupyter”. This will install the full Jupyter environment with full Python support. To add Julia Jupyter support, you need to run Julia another way (like just entering julia to get the REPL) and type “Pkg.add(“IJulia”)”. Now next time you start Jupyter (usually by typing “jupyter notebook”), you can create a new notebook based on Julia rather than Python.

Julia Packages

Once you have the core Julia programming environment up and running, you will probably want to install a number of add-on packages. The package manager is call Pkg and you need to type “using Pkg” before using it. These are all installed by the Pkg.add(“”) command. You only need to add a package once. You will probably want to run “Pkg.update()” now and again to see if the packages you are using have been updated.

There are currently about 1900 registered Julia packages. Not all of them have been updated to Julia version 1.0 yet, so check the compatibility first. There are a lot of useful packages for things like machine learning, scientific calculations, data frames, plotting, etc. Certainly have a look at the package library before embarking on writing something from scratch.

Summary

These are currently the main ways to play with Julia. I’m sure since Julia is a very open community driven system, that these will proliferate. I don’t miss using the giant IDEs Visual Studio or Eclipse, these have become far too heavy and slow in my opinion. I find I evenly distribute my time between using Jupyter, Juno and just edit/run. Compared to Python it may appear their aren’t nearly as many tools for Julia, but with the current set, I don’t feel deprived.

 

Written by smist08

October 10, 2018 at 3:55 am