Babun shell

Babun is linux kind of shell to work in windows platform. I first came to know about babun from zeroturnaround article. You can read about babun more here. Here are the list of features which

a) scp causes some trouble when working with cygwin. scp works well with babun. The experience is the same as if the machine is Linux. Babun also helps auto completing the hostname (pressing tab enables this) for the 2nd and subsequent times when doing ssh/scp.

b) To run the previous command which started with, for example, xyz, we run !xyz in linux. The drawback is that we don't know what command will run till the command gets executed (at least this happens in sles machines). But in babun after we type !xy and pressing tab it displays what was the command that was last repeated which started with xy. Very much helpful.

c) Babun also displays the working directory at the right hand side of the place where we type commands. It helps us to know in which is our current working directory without typing pwd.

d) Babun shell, by default, has huge amount of buffer size (99000 lines) which helps us to know all the commands that got executed from the time the shell got launched. This is not by default enabled in windows cmd. Max height size of windows cmd's screen buffer size is 9999 lines.

e) history command is also supported.

As babun is built on top of cygwin, it supports vi editor, find, cd -, grep, ls, pwd and other linux commands. 

There are some linux commands like ifconfig, which are not supported.

mvn install -pl with -am/-amd usage

mvn install -pl with -am/-amd is very powerful. I first came to know about this usage in this article.

One use case that I can think of is this. Assume we are making changes in project D. It depends on projects E, F, G, & H. Now assume we have done update of our local codebase from repo and it caused changes in all of the projects in our local codebase (to keep our scenario simple). Now to build project D, we can run "mvn install -pl <groupId:artifactId of project D> -am" which will make sure maven runs the E, F, G & H first before running our target project D.

-amd option is useful in the following scenario. Assume project X is where we are making changes and project A, B, C depend on X. Now we are making changes in project X, and we want to check whether project A, B, C are still compiling fine. Now we can run "mvn install -pl <groupId:artifactId of project X> -amd" which will do the job. 

These commands has to be run from the root of the code repo (or ... which I leave it as an exercise. It has something to do with lowest common denominator).

Junit and TestNG - An important difference to be noted

Assume the tests are run sequentially to keep things simple. Have a look at the following code:

a) Using Junit

import java.util.ArrayList;
import java.util.List;

import org.junit.Before;
import org.junit.Test;

public class JunitExample
{
    private List<Integer> ints = new ArrayList<>();
   
    @Before
    public void initialize()
    {
        ints.add(1);
        ints.add(2);
        ints.add(3);
    }

    @Test
    public void test1()
    {
        ints.add(4);
        System.out.println(ints);
    }
   
    @Test
    public void test2()
    {
        ints.add(5);
        System.out.println(ints);
    }
}

Running the tests sequentially outputs
[1, 2, 3, 4]
[1, 2, 3, 5]

b) Using TestNG

import java.util.ArrayList;
import java.util.List;

import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;


public class TestNGExample
{
    private List<Integer> ints = new ArrayList<>();
   
    @BeforeTest
    public void initialize()
    {
        ints.add(1);
        ints.add(2);
        ints.add(3);
    }

    @Test
    public void test1()
    {
        ints.add(4);
        System.out.println(ints);
    }
   
    @Test
    public void test2()
    {
        ints.add(4);
        System.out.println(ints);
    }
}

This outputs,
[1, 2, 3, 4]
[1, 2, 3, 4, 4]

Surprised?. This is because Junit creates a separate instance of test class for each and every tests that are present. When test1 begins to gets executed ints consisted of [1, 2, 3] and when test2 begins to gets executed too it consisted of [1, 2, 3]. No surprises here. We ended up with [1, 2, 3, 4] on both these cases.

But in case of TestNG, there is only one instance of test class that gets created and that will be used by all test methods. So, before test1 got executed, ints were [1, 2, 3] and by the time test1 got completed, ints were [1, 2, 3, 4]. Now when test2 begins to get executed ints are already [1, 2, 3, 4] and we add up 4 again to ints. So, finally we ended up with [1, 2, 3, 4, 4].

When the tests are run in parallel, in case of TestNG, the behavior is undefined if instance variables are used without considering multithreading scenarios.

svn checkout and update --depth/--set-depth option usage

Assume you have a project structure as this: You have a core-spoke model. The spokes depend on core piece. Different teams of developers are working with different sets of core and spoke projects. You have this project in SVN repository.

CodebaseRoot -> Core
                          -> Core1
                          -> Core2
                          -> .........
                          -> Core k
                       -> Spoke
                          -> Spoke1
                          -> Spoke2
                          -> ..........
                          -> Spoke n

Where each spoke component is dependent on some/all of the core components.

Assume this is a large codebase and you might need to work only with subset of spoke and core components. You need not checkout the whole codebase.

Selectively checking out the codebase can be done using the following steps: Be under the directory where you want to check out the code.

a) Run svn co --depth=immediates at <CodebaseRootURL>
    Now you have a local copy containing only Core and Spoke empty directories.

b) Run cd Core. Then run svn up --set-depth=infinity with <CodebaseRoot>/Core URL.
    This will checkout the core code piece completely. You can selectively checkout  too. For now let us assume that our spoke projects depend on all core pieces.

c) Run cd ../Spoke. Then run svn up --set-depth=immediates with <CodeBaseRoot>/Spoke URL. Now the set-depth=immediates option is used as this will checkout empty directories of all spoke directories.

d) Assume you are working only in project Spoke m. Run cd Spoke m. Then run svn up --set-depth=infinity with <CodeBaseRoot>/Spoke/Spoke m URL. This will checkout the whole spoke m project.

Hopefully this helps. 

System.setOut API in Java

System.out.println, by default, logs a message to console if the process has a reference to console window. For subprocesses, by default, there is no console attached. By making use of System.setOut we can redirect the output to a file or some other stream.

Code to redirect the output stream to a file.

File out = new File("/tmp/out.txt");
PrintStream outputStream = new PrintStream(out);
System.setOut(outputStream);

Now,
System.out.println("Hello world!");

This will result in Hello world! getting printed in /tmp/out.txt file.

Similarly, System.setIn and System.setErr APIs are also available and can be used wherever needed.

General Software Development blogs - Part 1

 
The ByteBaker - PhD student at Cornell University.

Coding Horror - programming and human factors

Martin Fowler Bliki

Mark Needham - Thoughs on Software Development

Google Testing Blog

Misko Hevery - The Testability Explorer Blog

Joel on Software

Object Mentor Articles

Scott Hanselman Programming 

The Database Programmer - All things related to database applications, both Desktop and Web

JVM related blogs

NoBlogDefFound - Tomasz Nurkiewicz around Java and code quality 

Vanilla #Java - Understanding how Core Java really works can help you write simpler, faster applications. 

Inspired by Actual Events - Dustin's Software Development Cogitations and Speculations 

The Takipi Blog - Server Debugging Made Easy 

ZeroTurnAround RebelLabs 

Java, SQL and jOOQ - Best Practices and Lessons Learned from Writing Awesome Java and SQL Code. Get some hands-on insight on what's behind developing jOOQ. 

Plumbr - Java Performance Tuning blog 

Vlad Mihalcea's Blog - Teaching is my way of learning 

Nicolas Fränkel blog - A Java Geek 

Petri Kainulainen - Do you want to be a better software developer? 

Stephen Colebourne's blog  - Thoughts and Musings on the world of Java and beyond 

How to do it in Java - Java Tutorials and Best Practices 

JaxEnter 

JavaSpecialists

Adam Bien's Weblog  

Java Concurrency - Jeremy Manson's blog, which goes into great detail either about concurrency in Java, or anything else that the author happens to feel is interesting or relevant to the target audience.

Links to learn about REST (REpresentational State Transfer) theoretically

Must go through links

 

1) Intro to REST by Google Developers YouTube link

The description of the video contains additional resources. Please check those links too.

 

2) How I Explained REST to My Wife by Ryan Tomayko

3) A Brief Introduction to REST

 

Additional Links

 

1) REST FAQ's by Jon Flanders

2a) Nobody Understands REST or HTTP

2b) Some People Understand REST and HTTP

 

3)  REST APIs must be hypertext-driven 
Under the comments section, Roy Fielding has answered lot of questions posted by users. Kindly go through it.

Java - Process, Subprocess and Deadlocks

We create subprocesses in Java using ProcessBuilder API. One glitch in that APIs are that there is a possibility of both the parent process and child process getting blocked or being deadlocked, if we forget to do some things. Process class Javadoc mentions about it.

Understanding about SubProcess' Default Output stream

Do you know what happens when System.out.println is called on a subprocess? It does not go to console. The subprocesses does not have console attached to it. Then where does the bytes go?

If you have played with Linux command terminal a bit, you would have known about pipes. 

Pipes are all about passing output of one command as input to other command. Now you would have asked a question to yourselves that there should be a storage where the output of the first command should have been saved. Yes, correct. There is a storage called buffer allocated by kernel for saving this information and the second command will read the input from that buffer.

Ex: ps -ef | grep java is equivalent to the ps -ef > kernel buffer < grep java.

There is a limitation with the above approach. The kernel buffer size is limited and if the output of the first command is big enough than kernel buffer then the command will get stuck.

This is the same logic that gets applied when we create subprocess using Java. The stuff that subprocess writes using System.out.println gets written to shared os buffer. We will get our hands dirty by simulating it.

SImulating the deadlock by filling the internal buffer

a) First create a Java Project and create a package called test.

b) Then create the MainProcess class and SubProcess class as given below.

c) MainProcess simply launches SubProcess class and waits for its completion.
To make sure the class is found in JVM's classpath, we need to set the directory where the JVM should search for the SubProcess class.

d) In the SubProcess class, we just run java command. You have to make sure that JAVA_HOME/bin is set in class path. Running java command just lists the help message. It is displayed in standard error stream. We just read from it and write to System.out.println.

e) As we write to System.out.println in subprocess, it gets written to the shared buffer. We are looping for 50 times and writing to the buffer to make sure that the buffer is filled and deadlock is simulated.

package test;

import java.io.File;
import java.io.IOException;

public class MainProcess
{
    public static void main(String[] args)
    {
        ProcessBuilder pb = new ProcessBuilder();
        pb.command("java", "test.SubProcess");

        try
        {
            pb.directory(new File(new File(".").getCanonicalFile(), "bin"));
            Process p = pb.start();
            p.waitFor();
        } catch (IOException | InterruptedException e)
        {
            e.printStackTrace();
        }
    }
}

package test;
 
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;

public class SubProcess
{
    public static void main(String[] args) throws IOException
    {
        List<String> commandOutput = getJavaCommandHelp();

        for (int i = 1; i <= 50; i++)
        {
            System.out.println(commandOutput);
        }
    }

    private static List<String> getJavaCommandHelp() throws IOException
    {
        Process p = new ProcessBuilder("java").start();
        List<String> allLines = new ArrayList<String>();
        InputStream is = p.getErrorStream();
        BufferedReader br = new BufferedReader(new InputStreamReader(is));

        String curLine;

        while ((curLine = br.readLine()) != null)
        {
            allLines.add(curLine);
        }

        return allLines;
    }
}

What is the fix

In the MainProcess after the line ProcessBuilder pb = new ProcessBuilder(), write the following line.

pb.inheritIO();

This line makes sure that whenever we write something using System.out.println in a subprocess, it gets written to output stream of MainProcess which in this case is Console. Now there won't be any deadlock.

Common mistakes which we do with Java Code - Part 2

1). Can you predict what the program will print?

public class Problem1

{

   public static void main(String[] args)

   {

      int[] arr = new int[] {1, 2, 3, 4, 5};

      for (int i = 0; i < arr.length; i++)

      {

          arr[i] = arr[i] * arr[i];

      }

      System.out.println(arr);

   }

}

Predicted Output:

[1, 4, 9, 16, 25]

Real output:

something like [I@5df9cdda

Analysis:

Array is an object. It’s toString is defined as “Returns a string consisting of the name of the class of which the object is an instance, the at-sign character @, and the unsigned hexadecimal representation of the hash code of the object.”

In our case the array is an int and arr.getClass() returns class [I and here the class name is [I , followed by @, followed by the hashcode of the array.

How to overcome this:

Use java.util.Arrays.toString, for printing 1D arrays. If the array contains objects which are instances of your application, then it is a must you need to override toString implementation. If not, in your logs you might see some output like the above one.

Make it a habit to override toString implementation which will also be useful during debugging of the Java programs.

2)  Can you predict what the program will print?

class Problem2

{

   public static void main (String[] args)

   {

      //0x01FF is the unicode representation of ǿ.

      char c = 0x01FF;

      byte[] bytes = String.valueOf(c).getBytes();

      System.out.println(java.util.Arrays.toString(bytes));

   }

}

Predicted output:

a) [-57, -65] b) [63] c) it depends d) none of the above

Expected output:

it depends. on utf 8 machines, it prints (a) and on win 7 32 bit machines it prints (b).

Analysis - What is the problem

Java, by default, takes the charset of the machine (where we run the program) when we call getBytes of the string, read/write to/from a Input/OutputStream, if we have not mentioned the charset explicitly. This causes different behavior across platforms.

How to overcome this

If any API has any overloaded method which takes encoding, go ahead and use this. It is recommended to explicitly mention the encoding/charset even if the string contains simple English characters. This makes the scope clearer for the reader about the details of the String. If we have not mentioned the encoding there are 2 possible cases. The string either contains simple English characters (or) the developer had forgotten to mention the encoding.

Recommended APIs

   

Till Java 6 there was a need for us to mention the encoding as a String. That is we need to hardcode “UTF-8” or “UTF-16”. I usually faced a doubt about which String is valid. That is “utf8” or “utf-8” or “UTF8” or “UTF-8” and so on. Every time there was a need for me to go to the internet to check which one is correct as there is some room for confusion. Thankfully in Java 7 we have  StandardCharSets. This makes it easy for the developers to write the correct code in the first place.

Files (has bunch of static methods to work with files).

Must Read Article

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

3) Can you predict what the program will print? (Taken from Java Puzzlers but slightly modified)

import java.math.BigDecimal;

class Problem3

{

   public static void main (String[] args)

   {

      BigDecimal costOfApple = new BigDecimal(0.3);

      BigDecimal costOfOrange = new BigDecimal(0.6);

      BigDecimal totalCost = costOfApple.add(costOfOrange);

      System.out.println(totalCost);

   }

}

Expected Output:

0.9

Real Output:

0.899999999999999966693309261245303787291049957275390625

Analysis:

We know that we should not use float/double for money calculations as it might not yield correct value. We have used BigDecimal yet we have hit an issue. First let us look at the doc of new BigDecimal(double). Point 1 explains in detail why we are seeing the issue. It says,

“new BigDecimal(0.1) in Java creates a BigDecimal which is exactly equal to 0.1 (an unscaled value of 1, with a scale of 1), but it is actually equal to 0.1000000000000000055511151231257827021181583404541015625”.

Only addition/multiplication combinations of (½) and its powers alone can be represented exactly. For example 0.5 can be represented exactly in the computers. To verify, you can write a program which prints new BigDecimal(0.5) which will print out 0.5, but new BigDecimal(0.2), new BigDecimal(0.4) won’t exactly print the same values.

Assigning double variable with value of 0.2, 0.4 will print these values exactly. This is because while printing the value, JVM takes care of printing only exact decimal digits to differentiate from the floating point number below from it and the floating point number above from it. They don’t print all the digits. BigDecimal exposes the problem clearly.

How to overcome this

By using BigDecimal’s String constructor. That is, new BigDecimal(“0.4”) will both store and print the value as 0.4.

References:

1. Engineering Disasters - Software flaws
2. first puzzle - Java Puzzlers - Google IO Tech Talks

Common mistakes which we do with Java Code - Part 1

1) Can you predict the output of the following code.

class Problem1
{
    public static void main(String[] args)
    {
        System.out.println(Boolean.getBoolean("true"));
    }
}

Predicted output:

true

Real output:

false

Analysis:

Boolean.getBoolean searches for that system property (WHAT!!!) and if that is equalIgnoreCase to "true" then only this will return true. In all cases this will return false. So, in this case the output is false.

How to overcome this?

As we use parseInt to convert String to int, parseFloat to convert String to float, we need to use parseBoolean.

System.out.println(Boolean.parseBoolean("true")) will return true.

2) What does the following program print?

class Problem2
{
    public static void main(String[] args)
    {
        long num = 10_000 * 10_000 * 10_000 * 10_000;
        System.out.println(num);
    }
}

Predicted output:

1 followed by 16 zeros.

Real output:

a negative value

Analysis:

Negative number output implies a overflow but howcome this can happen as I have used long to store this value?

Whenever we use a number, by default, it is assumed to be int. All the operations which we do on this number will be treated as int. So in this example by the time we do the 3rd multiplication the result will not fit in the int and returns a negative number.

We need to use L after the number to treat the number as long. One more glitch here is that the overflow will still happen if we put L as part of the last number. That is,
num = 10_000 * 10_000 * 10_000 * 10_000L will also overflow because by the time the compiler sees L, the result would have already overflowed.

How to overcome this?

Put L after the first number or introduce a 1L and multiply the rest of the numbers.

long num = 1L * 10_000 * 10_000 * 10_000 * 10_000;

will store num having 1 followed by 16 zeros.

3) What does the following program print?

class Problem3
{
    public static void main(String[] args)
    {
        java.util.List<Integer> ints = new java.util.ArrayList<>();
        for (int i = 1; i <= 5; i++)
        {
            ints.add(i);
        }
       
        ints.remove(3);
       
        System.out.println(ints);
    }
}

Predicted output:
[1, 2, 4, 5]

Real output:

[1, 2, 3, 5]

Analysis:

In this problem I add integers from 1 to 5 to the list and I remove 3 from the list. I expect it to print [1, 2, 4, 5]. Is this true? Think about it.

The problem is due to the introduction of autoboxing and unboxing features in JDK 5. In List I can add only Integer objects and not primitives. To make our life simpler, Java Compiler takes care of auto boxing the int value to Integer object. So, after the end of the loop we have a list with contents [1, 2, 3, 4, 5]. True. But remove is confusing. If we look at the documentation, we have 2 remove methods. One removes based on the index and other removes Objects. In our case which one will the compiler pick? As we have a method which takes an int, there is an exact match to call that method. The value 3 is treated as primitive itself and it is not autoboxed to Integer object 3. The effect is that we remove the value present at the index 3 which is 4.

How to overcome this?

ints.remove((Object) 3) will make the compiler call the remove(Object obj) method and this will return [1, 2, 4, 5].

SPOJ - Enormous Input and Enormous Output - Solution in Java

Today we will discuss about how to go about solving
ENORMOUS INPUT and ENORMOUS INPUT and OUTPUT problem.

The purpose of the first problem is read input as fast as possible. To be precise, we need to read 2.5MB/sec.

The second problem is to read input as fast as possible as well as output the results faster.

Problem 1 : Approach

Using BufferedReader to read the inputs.

The following solution contains nothing complicated to explain. 

a) We have used try with resources of Java 7 which takes care of closing the resource even in case of an exception. This is one of the best practices in Java 7.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
 
class EnormousInput
{
    public static void main(String[] args) throws IOException
    {
        int totalNumsDivisibleByK = 0;
      
        try(BufferedReader br = new BufferedReader(new InputStreamReader(System.in)))
        {
            int[] nAndK = getNandK(br);
            int n = nAndK[0];
            int k = nAndK[1];
            int[] allInputs = getAllInputs(br, n);
            totalNumsDivisibleByK = findTotalNumsDivisibleByK(allInputs, k);
        }
      
        System.out.print(totalNumsDivisibleByK);
    }

    private static int[] getAllInputs(BufferedReader br, int n) throws IOException
    {
        int[] allIntegers = new int[n];
        for (int i = 0; i < allIntegers.length; i++)
        {
            allIntegers[i] = Integer.parseInt(br.readLine());
        }
      
        return allIntegers;
    }

    private static int findTotalNumsDivisibleByK(int[] allInputs, int k) throws IOException
    {
        int totalNumsDivisibleByK = 0;
      
        for (int i = 0; i < allInputs.length; i++)
        {
            if (allInputs[i] % k == 0)
            {
                totalNumsDivisibleByK++;
            }
        }
        return totalNumsDivisibleByK;
    }

    private static int[] getNandK(BufferedReader br) throws IOException
    {
        int[] aNandK = new int[2];
      
        String firstLine = br.readLine();
        String[] tokens = firstLine.split("\\s+");
        int n = Integer.parseInt(tokens[0]);
        int k = Integer.parseInt(tokens[1]);
        aNandK[0] = n;
        aNandK[1] = k;
      
        return aNandK;
    }
}

This solution takes around 0.90 secs and gets accepted.

Problem 2 : Approach

Here also we use BufferedReader to read the inputs. For printing outputs we need to have a similar mechanism so that we don't spend much time in the I/O. We can use BufferedWriter. I have tried with StringBuilder to accumulate the outputs and print the final output once. 

Each outputs must be seperated by new line so I have made use of System.lineSeparator() which is available in Java 7.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
 
class EnormousInputAndOutput
{
    public static void main(String[] args) throws IOException
    {
        try(BufferedReader br = new BufferedReader(new InputStreamReader(System.in)))
        {
            int n = Integer.parseInt(br.readLine());
           
            String output = getMultipliedValueOfEachInput(br, n);
           
            System.out.print(output);
        }
    }

    private static String getMultipliedValueOfEachInput(BufferedReader br, int n) throws IOException
    {
        StringBuilder sb = new StringBuilder();
       
        for (int i = 1; i <= n; i++)
        {
            String[] nums = br.readLine().split("\\s+");
            int num1 = Integer.parseInt(nums[0]);
            int num2 = Integer.parseInt(nums[1]);
           
            int output = num1 * num2;
           
            sb.append(output);
            sb.append(System.lineSeparator());
        }
       
        return sb.toString();
    }
}

The solution takes around 1.30 secs and gets accepted.