Apache Kafka Cheatsheet

Here I present cheatsheet composed of snippets used by me in everyday work with Apache Kafka.
I feel that this can be helpful in working with this cheerful technology 😉

Note:
Commands names and addresses/ports may differ depending on your infrastructure and way how you installed Kafka (e.g. you can have Docker based installation).

If you notice any lack / error then please let me know in the comment.


JVM – Aggregation of the most frequently used jvm parameters

Max/Start Heap Size

-Xmx2g
-Xms2g

Garbage Collector Algorithms

-XX:+UseSerialGC
-XX:+UseParallelGC
-XX:+UseParNewGC
-XX:+UseG1GC

Garbage Collector Logging

-XX:+UseGCLogFileRotation 
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=50M
-Xloggc:/path.log
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps

Heap dump on out of memory exception

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./path.hprof

Jmx configuration

No authentication

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8086
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

Password authentication

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8086
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true 
-Dcom.sun.management.jmxremote.password.file=./jmxremote.password 
-Dcom.sun.management.jmxremote.access.file=./jmxremote.access

http://www.baeldung.com/jvm-parameters
http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html


Oracle – How to find out pending transactions/sessions?

Find all transactions on instance

Sometimes you want to know what transactions are actually on your db instance(e.g. when db works slow).
You can do it using the following query:

SELECT s.taddr,
       s.sid,
       s.serial#,
       s.username,
       s.SCHEMANAME,
       s.machine,
       s.status,
       s.lockwait,
       t.start_time
  FROM v$transaction t
  INNER JOIN v$session s ON t.addr = s.taddr;

v$transaction contains data about each transaction existing on the instance when v$session contains all sessions.

As a result you will get all transactions on instance with info about corresponding sessions.

Find out which sessions blocks another sessions

Moreover you can check if any session wait for another. You can easy find it by blocking_sesssion column in v$session view.

SELECT sid blocked_session_id,
       seconds_in_wait,
       blocking_session blocking_session_id
  FROM v$session
  WHERE blocking_session IS NOT NULL
  ORDER BY blocking_session;

Find out on which sql statement session was blocked

To find out this sql you can join to previous query v$sql view by sql_id column in v$session.

SELECT s.sid blocked_session_id,
       s.seconds_in_wait,
       s.blocking_session blocking_session_id,
       sq.sql_fulltext
  FROM v$session s
  LEFT JOIN v$sql sq ON sq.sql_id = s.sql_id
  WHERE blocking_session IS NOT NULL
  ORDER BY blocking_session;

Links:

https://stackoverflow.com/questions/1299694/oracle-how-to-find-out-if-there-is-a-transaction-pending
http://www.dba-oracle.com/t_find_blocking_sessions.htm
Reference – V$SESSION
Reference – V$TRANSACTION
Reference – V$SQL


Linux – How to find files used by process?

Let’s assume that we have given id of process (PID) 6546 and we want to find out which files process actually uses.
Very important here is actually, because we don’t want all the files used by process so far.

To do it we can use proc pseudo file system.
Under /proc we can find directory for each process actually running on system.
Names of these directories are PIDs of processes.

In each of these directories we can find interesting data about the processes(file descriptors, command line arguments, environment variables).

In directory /proc/6546/fd we can find all files actually opened by process with pid 6546.

Links

http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html
http://man7.org/linux/man-pages/man5/proc.5.html
https://www.cyberciti.biz/faq/linux-pidof-command-examples-find-pid-of-program/


How to force Spring Boot Security to say a little bit more?

By default spring boot security isn’t too talkative.
If you struggle with some problems and want to see what is going on under the covers, you can always switch to wider log level.

You can do it by providing this property in your application.yml or application.properties file.

logging.level.org.springframework.security: DEBUG

Additionaly you can turn on Spring Web debug logs and set debug parameter to true in your @EnableWebSecurity annotation

logging.level.org.springframework.web: DEBUG
@EnableWebSecurity(debug = true)
public class SecurityConfig extends WebSecurityConfigurerAdapter {...}

How jsonpath can help you to test rest controllers?

Introduction

In this post, I will show how you can write assertions to json strings.
When I test my rest controllers, I want to check that responses bodies contains exactly the json that I assume.

Json path is a tool which can return you only the part of json that you want.
You give it a pattern to express which part of data you want to receive.

JsonPath patterns

I have the following json:

[
	{
		"name":"Lukas",
		"age":23
	},
	{
		"name":"Jack",
		"age":35
	},
	{
		"name":"Michael",
		"age":32
	}
]

When I use pattern:

$[0].name

I receive a string:
“Lukas”
When pattern will be:

$.[1].age

Result:
35

You can see that expressions are something like regexp for json.

Java tests examples

Ex.1
Lets assume that our controller should return json similar to this:

{
	“id”:12
}

when we request it by -> /user/12

@Test
public void shouldReturnUserWithId() throws Exception {
   String responseContent = mockMvc.perform(get("/user/" + USER_ID).accept(MediaType.APPLICATION_JSON_UTF8))
           .andExpect(status().isOk())
           .andReturn().getResponse().getContentAsString();

   Integer loginFromResponse = JsonPath.parse(responseContent).read("$.id");
   Assert.assertEquals(USER_ID, loginFromResponse.intValue());
}

Ex.2
We assume that json should be an object with array of objects with “id” property.

{
	[
{	
	“id”:1
},
{	
	“id”:2
},
{	
	“id”:3
}
	]
}
@Test
public void shouldReturnArrayOfUsers() throws Exception {
   String responseContent = mockMvc.perform(get("/user").accept(MediaType.APPLICATION_JSON_UTF8))
           .andExpect(status().isOk())
           .andReturn().getResponse().getContentAsString();

   List<integer> ids = JsonPath.parse(responseContent).read("$[*].id");
   MatcherAssert.assertThat(ids, Matchers.hasSize(3));
   MatcherAssert.assertThat(ids, Matchers.containsInAnyOrder(1,2,3));
}
</integer>

As you can see, we can use hamcrest mathers to test collection returned by jsonpath api.

Ex.3

@Test
public void shouldReturnsUsersWithListsOfPosts() throws Exception {
   String responseContent = mockMvc.perform(get("/user").accept(MediaType.APPLICATION_JSON_UTF8))
		   .andExpect(status().isOk())
		   .andReturn().getResponse().getContentAsString();

   List<string> postTitles = JsonPath.parse(responseContent).read("$[*].posts[0].title");
   MatcherAssert.assertThat(postTitles, Matchers.hasSize(3));
   MatcherAssert.assertThat(postTitles, Matchers.everyItem(Matchers.is("post1")));
}
</string>

Here is shown that you can easily extract repeatedly nestet items.

Tools

To test your jsonpath expression you can testers which are many on the internet.
Here you can find one of them:
https://jsonpath.curiousconcept.com/

Links

https://github.com/jayway/JsonPath
https://github.com/raphaelsolarski/jsonpath-to-test-rest-controller-example


Hamcrest collection matchers – learn by examples.

Hello.
In this post I will show how you can improve quality and clarity of your assertions using matchers in context of testing collections.

import com.google.common.collect.Lists;
import org.junit.Test;

import java.util.List;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.*;

public class HamcrestCollectionsLT {

    @Test
    public void collectionShouldHasOneGivenItem() throws Exception {
        List<String> list = Lists.newArrayList("Foo");
        assertThat(list, contains("Foo"));
    }

    @Test
    public void collectionShouldHasExactlyGivenItemsInGivenOrder() throws Exception {
        List<String> list = Lists.newArrayList("Foo", "Bar");
        assertThat(list, contains("Foo", "Bar"));
    }

    @Test
    public void collectionShouldHasExactlyGivenItemsInAnyOrder() throws Exception {
        List<String> list = Lists.newArrayList("Foo", "Bar");
        assertThat(list, containsInAnyOrder("Bar", "Foo"));
    }

    @Test
    public void collectionShouldHasGivenItem() throws Exception {
        List<String> list = Lists.newArrayList("Foo", "Bar", "Element");
        assertThat(list, hasItem("Foo"));
    }

    @Test
    public void collectionShouldNotHasGivenItem() throws Exception {
        List<String> list = Lists.newArrayList("Bar", "Element");
        assertThat(list, not(hasItem("Foo")));
    }

    @Test
    public void collectionShouldBeEmpty() throws Exception {
        List<String> list = Lists.newArrayList();
        assertThat(list, empty());
    }

    @Test
    public void collectionShouldNotBeEmpty() throws Exception {
        List<String> list = Lists.newArrayList("Foo");
        assertThat(list, not(empty()));
    }

    @Test
    public void collectionShouldHasGivenLength() throws Exception {
        List<String> list = Lists.newArrayList("Foo", "Bar", "Element");
        assertThat(list, hasSize(3));
    }

    @Test
    public void collectionShouldNotHaveGivenLength() throws Exception {
        List<String> list = Lists.newArrayList("Foo");
        assertThat(list, not(hasSize(2)));
    }

    @Test
    public void everyCollectionItemShouldBeGreater() throws Exception {
        List<Integer> list = Lists.newArrayList(4, 5, 6);
        assertThat(list, everyItem(greaterThan(3)));
    }

    @Test
    public void everyCollectionItemShouldBeGreaterOrEqual() throws Exception {
        List<Integer> list = Lists.newArrayList(4, 5, 6);
        assertThat(list, everyItem(greaterThanOrEqualTo(4)));
    }

    @Test
    public void everyCollectionStringItemContainSubstring() throws Exception {
        List<String> list = Lists.newArrayList("Sub1", "Sub2", "Sub3");
        assertThat(list, everyItem(containsString("Sub")));
    }

    @Test
    public void shouldContainAtLastGivenElementsAndEveryElementShouldContainGivenString() throws Exception {
        List<String> list = Lists.newArrayList("Sub1", "Sub2", "Sub3");
        assertThat(list, allOf(hasItems("Sub1", "Sub2"), everyItem(containsString("Sub"))));
    }

}

Links:
https://github.com/hamcrest/JavaHamcrest


Spring test configuration example.

Hello.
Today I want to show you simple test configuration in spring mvc project.
In example I use java based configuration(I admit that this kind of configuration I prefer).
But in future I want to present xml based configuration, too.

On the bottom of the post you find link to github repository with full project.

Class under testing:

package com.raphaelsolarski.springtestconfig.controller;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

@Controller
public class MyController {

    @RequestMapping(path = "/", method = RequestMethod.GET)
    String home() {
        return "home";
    }

}

Test:

package com.raphaelsolarski.springtestconfig.controller;

import com.raphaelsolarski.springtestconfig.config.WebConfig;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.TestExecutionListeners;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;
import org.springframework.test.context.web.AnnotationConfigWebContextLoader;
import org.springframework.test.context.web.WebAppConfiguration;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.result.MockMvcResultMatchers;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import org.springframework.web.context.WebApplicationContext;

import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = {WebConfig.class}, loader = AnnotationConfigWebContextLoader.class)
@TestExecutionListeners(listeners = {DependencyInjectionTestExecutionListener.class})
@WebAppConfiguration
public class MyControllerTest {

    @Autowired
    private WebApplicationContext webApplicationContext;

    private MockMvc mockMvc;

    @Before
    public void setUp() {
        mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext).build();
    }

    @Test
    public void getRootShouldReturnHomeView() throws Exception {
        mockMvc.perform(get("/")).andExpect(MockMvcResultMatchers.view().name("home"));
    }

}

github-icon Full code on github.com

Links:
http://docs.spring.io/spring/docs/current/spring-framework-reference/html/integration-testing.html


How to compile maven project without web.xml?

When you use servlet api beyond 3.0 version, you can use java based configuration of your webapp instead of web.xml file.
Maven by default report an error when you try to compile project that does not contain web.xml file, so you have to additionally configure maven-war-plugin in pom.xml.

<plugin>
    <groupid>org.apache.maven.plugins</groupid>
    <artifactid>maven-war-plugin</artifactid>
    <version>2.2</version>
    <configuration>
        <failonmissingwebxml>false</failonmissingwebxml>
    </configuration>
</plugin>