Apache Kafka Cheatsheet
Here I present cheatsheet composed of snippets used by me in everyday work with Apache Kafka.
I feel that this can be helpful in working with this cheerful technology 😉
Note:
Commands names and addresses/ports may differ depending on your infrastructure and way how you installed Kafka (e.g. you can have Docker based installation).
If you notice any lack / error then please let me know in the comment.
JVM – Aggregation of the most frequently used jvm parameters
Max/Start Heap Size
-Xmx2g -Xms2g
Garbage Collector Algorithms
-XX:+UseSerialGC -XX:+UseParallelGC -XX:+UseParNewGC -XX:+UseG1GC
Garbage Collector Logging
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M -Xloggc:/path.log -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
Heap dump on out of memory exception
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./path.hprof
Jmx configuration
No authentication
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8086 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
Password authentication
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8086 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=./jmxremote.password -Dcom.sun.management.jmxremote.access.file=./jmxremote.access
http://www.baeldung.com/jvm-parameters
http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html
Oracle – How to find out pending transactions/sessions?
Find all transactions on instance
Sometimes you want to know what transactions are actually on your db instance(e.g. when db works slow).
You can do it using the following query:
SELECT s.taddr, s.sid, s.serial#, s.username, s.SCHEMANAME, s.machine, s.status, s.lockwait, t.start_time FROM v$transaction t INNER JOIN v$session s ON t.addr = s.taddr;
v$transaction contains data about each transaction existing on the instance when v$session contains all sessions.
As a result you will get all transactions on instance with info about corresponding sessions.
Find out which sessions blocks another sessions
Moreover you can check if any session wait for another. You can easy find it by blocking_sesssion column in v$session view.
SELECT sid blocked_session_id, seconds_in_wait, blocking_session blocking_session_id FROM v$session WHERE blocking_session IS NOT NULL ORDER BY blocking_session;
Find out on which sql statement session was blocked
To find out this sql you can join to previous query v$sql view by sql_id column in v$session.
SELECT s.sid blocked_session_id, s.seconds_in_wait, s.blocking_session blocking_session_id, sq.sql_fulltext FROM v$session s LEFT JOIN v$sql sq ON sq.sql_id = s.sql_id WHERE blocking_session IS NOT NULL ORDER BY blocking_session;
Links:
https://stackoverflow.com/questions/1299694/oracle-how-to-find-out-if-there-is-a-transaction-pending
http://www.dba-oracle.com/t_find_blocking_sessions.htm
Reference – V$SESSION
Reference – V$TRANSACTION
Reference – V$SQL
Linux – How to find files used by process?
Let’s assume that we have given id of process (PID) 6546 and we want to find out which files process actually uses.
Very important here is actually, because we don’t want all the files used by process so far.
To do it we can use proc pseudo file system.
Under /proc we can find directory for each process actually running on system.
Names of these directories are PIDs of processes.
In each of these directories we can find interesting data about the processes(file descriptors, command line arguments, environment variables).
In directory /proc/6546/fd we can find all files actually opened by process with pid 6546.
Links
http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html
http://man7.org/linux/man-pages/man5/proc.5.html
https://www.cyberciti.biz/faq/linux-pidof-command-examples-find-pid-of-program/
How to force Spring Boot Security to say a little bit more?
By default spring boot security isn’t too talkative.
If you struggle with some problems and want to see what is going on under the covers, you can always switch to wider log level.
You can do it by providing this property in your application.yml or application.properties file.
logging.level.org.springframework.security: DEBUG
Additionaly you can turn on Spring Web debug logs and set debug parameter to true in your @EnableWebSecurity annotation
logging.level.org.springframework.web: DEBUG
@EnableWebSecurity(debug = true) public class SecurityConfig extends WebSecurityConfigurerAdapter {...}
How jsonpath can help you to test rest controllers?
Introduction
In this post, I will show how you can write assertions to json strings.
When I test my rest controllers, I want to check that responses bodies contains exactly the json that I assume.
Json path is a tool which can return you only the part of json that you want.
You give it a pattern to express which part of data you want to receive.
JsonPath patterns
I have the following json:
[ { "name":"Lukas", "age":23 }, { "name":"Jack", "age":35 }, { "name":"Michael", "age":32 } ]
When I use pattern:
$[0].name
I receive a string:
“Lukas”
When pattern will be:
$.[1].age
Result:
35
You can see that expressions are something like regexp for json.
Java tests examples
Ex.1
Lets assume that our controller should return json similar to this:
{ “id”:12 }
when we request it by -> /user/12
@Test public void shouldReturnUserWithId() throws Exception { String responseContent = mockMvc.perform(get("/user/" + USER_ID).accept(MediaType.APPLICATION_JSON_UTF8)) .andExpect(status().isOk()) .andReturn().getResponse().getContentAsString(); Integer loginFromResponse = JsonPath.parse(responseContent).read("$.id"); Assert.assertEquals(USER_ID, loginFromResponse.intValue()); }
Ex.2
We assume that json should be an object with array of objects with “id” property.
{ [ { “id”:1 }, { “id”:2 }, { “id”:3 } ] }
@Test public void shouldReturnArrayOfUsers() throws Exception { String responseContent = mockMvc.perform(get("/user").accept(MediaType.APPLICATION_JSON_UTF8)) .andExpect(status().isOk()) .andReturn().getResponse().getContentAsString(); List<integer> ids = JsonPath.parse(responseContent).read("$[*].id"); MatcherAssert.assertThat(ids, Matchers.hasSize(3)); MatcherAssert.assertThat(ids, Matchers.containsInAnyOrder(1,2,3)); } </integer>
As you can see, we can use hamcrest mathers to test collection returned by jsonpath api.
Ex.3
@Test public void shouldReturnsUsersWithListsOfPosts() throws Exception { String responseContent = mockMvc.perform(get("/user").accept(MediaType.APPLICATION_JSON_UTF8)) .andExpect(status().isOk()) .andReturn().getResponse().getContentAsString(); List<string> postTitles = JsonPath.parse(responseContent).read("$[*].posts[0].title"); MatcherAssert.assertThat(postTitles, Matchers.hasSize(3)); MatcherAssert.assertThat(postTitles, Matchers.everyItem(Matchers.is("post1"))); } </string>
Here is shown that you can easily extract repeatedly nestet items.
Tools
To test your jsonpath expression you can testers which are many on the internet.
Here you can find one of them:
https://jsonpath.curiousconcept.com/
Links
https://github.com/jayway/JsonPath
https://github.com/raphaelsolarski/jsonpath-to-test-rest-controller-example
Hamcrest collection matchers – learn by examples.
Hello.
In this post I will show how you can improve quality and clarity of your assertions using matchers in context of testing collections.
import com.google.common.collect.Lists; import org.junit.Test; import java.util.List; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.*; public class HamcrestCollectionsLT { @Test public void collectionShouldHasOneGivenItem() throws Exception { List<String> list = Lists.newArrayList("Foo"); assertThat(list, contains("Foo")); } @Test public void collectionShouldHasExactlyGivenItemsInGivenOrder() throws Exception { List<String> list = Lists.newArrayList("Foo", "Bar"); assertThat(list, contains("Foo", "Bar")); } @Test public void collectionShouldHasExactlyGivenItemsInAnyOrder() throws Exception { List<String> list = Lists.newArrayList("Foo", "Bar"); assertThat(list, containsInAnyOrder("Bar", "Foo")); } @Test public void collectionShouldHasGivenItem() throws Exception { List<String> list = Lists.newArrayList("Foo", "Bar", "Element"); assertThat(list, hasItem("Foo")); } @Test public void collectionShouldNotHasGivenItem() throws Exception { List<String> list = Lists.newArrayList("Bar", "Element"); assertThat(list, not(hasItem("Foo"))); } @Test public void collectionShouldBeEmpty() throws Exception { List<String> list = Lists.newArrayList(); assertThat(list, empty()); } @Test public void collectionShouldNotBeEmpty() throws Exception { List<String> list = Lists.newArrayList("Foo"); assertThat(list, not(empty())); } @Test public void collectionShouldHasGivenLength() throws Exception { List<String> list = Lists.newArrayList("Foo", "Bar", "Element"); assertThat(list, hasSize(3)); } @Test public void collectionShouldNotHaveGivenLength() throws Exception { List<String> list = Lists.newArrayList("Foo"); assertThat(list, not(hasSize(2))); } @Test public void everyCollectionItemShouldBeGreater() throws Exception { List<Integer> list = Lists.newArrayList(4, 5, 6); assertThat(list, everyItem(greaterThan(3))); } @Test public void everyCollectionItemShouldBeGreaterOrEqual() throws Exception { List<Integer> list = Lists.newArrayList(4, 5, 6); assertThat(list, everyItem(greaterThanOrEqualTo(4))); } @Test public void everyCollectionStringItemContainSubstring() throws Exception { List<String> list = Lists.newArrayList("Sub1", "Sub2", "Sub3"); assertThat(list, everyItem(containsString("Sub"))); } @Test public void shouldContainAtLastGivenElementsAndEveryElementShouldContainGivenString() throws Exception { List<String> list = Lists.newArrayList("Sub1", "Sub2", "Sub3"); assertThat(list, allOf(hasItems("Sub1", "Sub2"), everyItem(containsString("Sub")))); } }
Spring test configuration example.
Hello.
Today I want to show you simple test configuration in spring mvc project.
In example I use java based configuration(I admit that this kind of configuration I prefer).
But in future I want to present xml based configuration, too.
On the bottom of the post you find link to github repository with full project.
Class under testing:
package com.raphaelsolarski.springtestconfig.controller; import org.springframework.stereotype.Controller; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; @Controller public class MyController { @RequestMapping(path = "/", method = RequestMethod.GET) String home() { return "home"; } }
Test:
package com.raphaelsolarski.springtestconfig.controller; import com.raphaelsolarski.springtestconfig.config.WebConfig; import org.junit.Before; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.TestExecutionListeners; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.support.DependencyInjectionTestExecutionListener; import org.springframework.test.context.web.AnnotationConfigWebContextLoader; import org.springframework.test.context.web.WebAppConfiguration; import org.springframework.test.web.servlet.MockMvc; import org.springframework.test.web.servlet.result.MockMvcResultMatchers; import org.springframework.test.web.servlet.setup.MockMvcBuilders; import org.springframework.web.context.WebApplicationContext; import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(classes = {WebConfig.class}, loader = AnnotationConfigWebContextLoader.class) @TestExecutionListeners(listeners = {DependencyInjectionTestExecutionListener.class}) @WebAppConfiguration public class MyControllerTest { @Autowired private WebApplicationContext webApplicationContext; private MockMvc mockMvc; @Before public void setUp() { mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext).build(); } @Test public void getRootShouldReturnHomeView() throws Exception { mockMvc.perform(get("/")).andExpect(MockMvcResultMatchers.view().name("home")); } }
Links:
http://docs.spring.io/spring/docs/current/spring-framework-reference/html/integration-testing.html
How to compile maven project without web.xml?
When you use servlet api beyond 3.0 version, you can use java based configuration of your webapp instead of web.xml file.
Maven by default report an error when you try to compile project that does not contain web.xml file, so you have to additionally configure maven-war-plugin in pom.xml.
<plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-war-plugin</artifactid> <version>2.2</version> <configuration> <failonmissingwebxml>false</failonmissingwebxml> </configuration> </plugin>