OSH/OKD CLI Cheatsheet

Login

oc login -u 'foo-user' https://foo-cloud:443

Help

oc help

Show all projects

oc projects

Show current project

oc project

Switch to project

oc project foo-project

Show overview of current project

oc status

Get all resources by label selector

oc get all -l foo=bar -n foo-project

Show list of supported resources

oc api-resources

Get pods/routes/deploymentconfigs

oc get pods
oc get services
oc get routes
oc get deploymentconfigs

Describe given resource

oc describe deploymentconfigs foo-dc
oc describe pod foo-pod-xfdsbc

Connect by ssh to pod

oc rsh <pod-id>

Start and connect to pod for debugging

oc debug dc/some-app

Get logs from pod and follow

oc logs -f <pod-id>

Get logs from given container in pod (e.g. from some init-container)

oc logs <pod-id> -c <container-name>

Get logs from previous pod

oc logs -p <pod-id>

Get specified number of last logs from pod

oc logs --tail=5 <pod-id>

Delete by label (e.g. whole stack for given app)

oc delete all -l app=test

Port forwarding (for development)

oc port-forward <pod-name> <local-port>:<remote:port>


Apache Kafka Cheatsheet

Here I present cheatsheet composed of snippets used by me in everyday work with Apache Kafka.
I feel that this can be helpful in working with this cheerful technology 😉

Note:
Commands names and addresses/ports may differ depending on your infrastructure and way how you installed Kafka (e.g. you can have Docker based installation).

create topic

kafka-topics --create --zookeeper localhost:2181 --topic foo-topic --replication-factor 1 --partitions 1

list topics

kafka-topics --list --zookeeper localhost:2181

describe topics

kafka-topics --describe --zookeeper localhost:2181

change topic config

kafka-topics --zookeeper localhost:2181 --alter --topic some-topic --partitions 40

change topic retention

kafka-topics --zookeeper localhost:2181 --alter --topic some-topic --config retention.ms=432000000

list consumer groups

kafka-consumer-groups --bootstrap-server localhost:9092 --list

list consumer groups including old fashion zookeeper consumer groups

kafka-consumer-groups --zookeeper localhost:2181 --list

describe offsets

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group SomeConsumerGroup

write to topic

kafka-console-producer --topic some-topic --broker-list localhost:9092

write to topic with compression

kafka-console-producer --topic some-topic --broker-list localhost:9092

read from topic

kafka-console-consumer --bootstrap-server localhost:9092 --topic some-topic --from-beginning

read from topic in avro

kafka-avro-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic some-topic

shift topic offset for consumer group (preview)

kafka-consumer-groups --bootstrap-server localhost:9092  --group SomeConsumerGropu --shift-by -4 --reset-offsets --topic some-topic

shift topic offset for consumer group (real execution)

kafka-consumer-groups --bootstrap-server localhost:9092  --group SomeConsumerGropu --shift-by -4 --reset-offsets --topic some-topic --execute

shift topic offset for consumer group to given date (preview)

kafka-consumer-groups --bootstrap-server localhost:9092  --group SomeConsumerGroupt --to-datetime '2019-02-02T00:00:00.000' --reset-offsets --topic some-topic

shift topic offsets for consumer group to earliest

kafka-consumer-groups --bootstrap-server localhost:9092 --group SomeGroput --reset-offsets --to-earliest --topic some-topic --execute

shift topic offsets for consumer group to specific offset

kafka-consumer-groups --bootstrap-server localhost:9092 --group SomeGroput --reset-offsets --to-offset 18406 --topic some-topic --execute

how to manually create consumer group in console

kafka-console-consumer --bootstrap-server localhost:9092 --topic some-topic --group SomeConsumerGroup

save messages to file

kafka-console-consumer --bootstrap-server localhost:9092 --topic some-topic --from-beginning > messages.txt

find topic files on disk

find /var/lib/kafka/ -maxdepth 1 -type d -name '_schema*'

mark topic for deletion

kafka-topics --delete --zookeeper localhost:2181 --topic some-topic

mark all topics to delete

#!/bin/bash

TOPIC_NAMES=$(kafka-topics --zookeeper localhost:2181 --list)

for TOPIC in $TOPIC_NAMES
do
    kafka-topics --zookeeper localhost:2181 --delete --topic $TOPIC
done

trim topic using kafka retention

kafka-topics --zookeeper localhost:2181 --alter --topic _schemas --config retention.ms=1000
kafka-topics --zookeeper localhost:2181 --alter --topic _schemas --config cleanup.policy=delete
# wait and check if topic is empty
# reset to previous settings
kafka-topics --zookeeper localhost:2181 --alter --topic _schemas --delete-config retention.ms
kafka-topics --zookeeper localhost:2181 --alter --topic _schemas --config cleanup.policy=compact

read segment file

kafka-run-class kafka.tools.DumpLogSegments --deep-iteration --print-data-log --files 00000000000000000000.log

If you notice any lack / error then please let me know in the comment.


JVM – Aggregation of the most frequently used jvm parameters

Max/Start Heap Size

  1. -Xmx2g
  2. -Xms2g

Garbage Collector Algorithms

  1. -XX:+UseSerialGC
  2. -XX:+UseParallelGC
  3. -XX:+UseParNewGC
  4. -XX:+UseG1GC

Garbage Collector Logging

  1. -XX:+UseGCLogFileRotation
  2. -XX:NumberOfGCLogFiles=10
  3. -XX:GCLogFileSize=50M
  4. -Xloggc:/path.log
  5. -XX:+PrintGCTimeStamps
  6. -XX:+PrintGCDateStamps

Heap dump on out of memory exception

  1. -XX:+HeapDumpOnOutOfMemoryError
  2. -XX:HeapDumpPath=./path.hprof

Jmx configuration

No authentication

  1. -Dcom.sun.management.jmxremote
  2. -Dcom.sun.management.jmxremote.port=8086
  3. -Dcom.sun.management.jmxremote.ssl=false
  4. -Dcom.sun.management.jmxremote.authenticate=false

Password authentication

  1. -Dcom.sun.management.jmxremote
  2. -Dcom.sun.management.jmxremote.port=8086
  3. -Dcom.sun.management.jmxremote.ssl=false
  4. -Dcom.sun.management.jmxremote.authenticate=true
  5. -Dcom.sun.management.jmxremote.password.file=./jmxremote.password
  6. -Dcom.sun.management.jmxremote.access.file=./jmxremote.access

http://www.baeldung.com/jvm-parameters
http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html


Oracle – How to find out pending transactions/sessions?

Find all transactions on instance

Sometimes you want to know what transactions are actually on your db instance(e.g. when db works slow).
You can do it using the following query:

  1. SELECT s.taddr,
  2. s.sid,
  3. s.serial#,
  4. s.username,
  5. s.SCHEMANAME,
  6. s.machine,
  7. s.status,
  8. s.lockwait,
  9. t.start_time
  10. FROM v$transaction t
  11. INNER JOIN v$session s ON t.addr = s.taddr;

v$transaction contains data about each transaction existing on the instance when v$session contains all sessions.

As a result you will get all transactions on instance with info about corresponding sessions.

Find out which sessions blocks another sessions

Moreover you can check if any session wait for another. You can easy find it by blocking_sesssion column in v$session view.

  1. SELECT sid blocked_session_id,
  2. seconds_in_wait,
  3. blocking_session blocking_session_id
  4. FROM v$session
  5. WHERE blocking_session IS NOT NULL
  6. ORDER BY blocking_session;

Find out on which sql statement session was blocked

To find out this sql you can join to previous query v$sql view by sql_id column in v$session.

  1. SELECT s.sid blocked_session_id,
  2. s.seconds_in_wait,
  3. s.blocking_session blocking_session_id,
  4. sq.sql_fulltext
  5. FROM v$session s
  6. LEFT JOIN v$sql sq ON sq.sql_id = s.sql_id
  7. WHERE blocking_session IS NOT NULL
  8. ORDER BY blocking_session;

Links:

https://stackoverflow.com/questions/1299694/oracle-how-to-find-out-if-there-is-a-transaction-pending
http://www.dba-oracle.com/t_find_blocking_sessions.htm
Reference – V$SESSION
Reference – V$TRANSACTION
Reference – V$SQL


Linux – How to find files used by process?

Let’s assume that we have given id of process (PID) 6546 and we want to find out which files process actually uses.
Very important here is actually, because we don’t want all the files used by process so far.

To do it we can use proc pseudo file system.
Under /proc we can find directory for each process actually running on system.
Names of these directories are PIDs of processes.

In each of these directories we can find interesting data about the processes(file descriptors, command line arguments, environment variables).

In directory /proc/6546/fd we can find all files actually opened by process with pid 6546.

Links

http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html
http://man7.org/linux/man-pages/man5/proc.5.html
https://www.cyberciti.biz/faq/linux-pidof-command-examples-find-pid-of-program/


How to force Spring Boot Security to say a little bit more?

By default spring boot security isn’t too talkative.
If you struggle with some problems and want to see what is going on under the covers, you can always switch to wider log level.

You can do it by providing this property in your application.yml or application.properties file.

  1. logging.level.org.springframework.security: DEBUG

Additionaly you can turn on Spring Web debug logs and set debug parameter to true in your @EnableWebSecurity annotation

  1. logging.level.org.springframework.web: DEBUG
  1. @EnableWebSecurity(debug = true)
  2. public class SecurityConfig extends WebSecurityConfigurerAdapter {...}

How jsonpath can help you to test rest controllers?

Introduction

In this post, I will show how you can write assertions to json strings.
When I test my rest controllers, I want to check that responses bodies contains exactly the json that I assume.

Json path is a tool which can return you only the part of json that you want.
You give it a pattern to express which part of data you want to receive.

JsonPath patterns

I have the following json:

  1. [
  2. {
  3. "name":"Lukas",
  4. "age":23
  5. },
  6. {
  7. "name":"Jack",
  8. "age":35
  9. },
  10. {
  11. "name":"Michael",
  12. "age":32
  13. }
  14. ]

When I use pattern:

  1. $[0].name

I receive a string:
“Lukas”
When pattern will be:

  1. $.[1].age

Result:
35

You can see that expressions are something like regexp for json.

Java tests examples

Ex.1
Lets assume that our controller should return json similar to this:

  1. {
  2. id”:12
  3. }
  4.  

when we request it by -> /user/12

  1. @Test
  2. public void shouldReturnUserWithId() throws Exception {
  3. String responseContent = mockMvc.perform(get("/user/" + USER_ID).accept(MediaType.APPLICATION_JSON_UTF8))
  4. .andExpect(status().isOk())
  5. .andReturn().getResponse().getContentAsString();
  6.  
  7. Integer loginFromResponse = JsonPath.parse(responseContent).read("$.id");
  8. Assert.assertEquals(USER_ID, loginFromResponse.intValue());
  9. }
  10.  

Ex.2
We assume that json should be an object with array of objects with “id” property.

  1. {
  2. [
  3. {
  4. id”:1
  5. },
  6. {
  7. id”:2
  8. },
  9. {
  10. id”:3
  11. }
  12. ]
  13. }
  1. @Test
  2. public void shouldReturnArrayOfUsers() throws Exception {
  3. String responseContent = mockMvc.perform(get("/user").accept(MediaType.APPLICATION_JSON_UTF8))
  4. .andExpect(status().isOk())
  5. .andReturn().getResponse().getContentAsString();
  6.  
  7. List<integer> ids = JsonPath.parse(responseContent).read("$[*].id");
  8. MatcherAssert.assertThat(ids, Matchers.hasSize(3));
  9. MatcherAssert.assertThat(ids, Matchers.containsInAnyOrder(1,2,3));
  10. }
  11. </integer>

As you can see, we can use hamcrest mathers to test collection returned by jsonpath api.

Ex.3

  1. @Test
  2. public void shouldReturnsUsersWithListsOfPosts() throws Exception {
  3. String responseContent = mockMvc.perform(get("/user").accept(MediaType.APPLICATION_JSON_UTF8))
  4. .andExpect(status().isOk())
  5. .andReturn().getResponse().getContentAsString();
  6.  
  7. List<string> postTitles = JsonPath.parse(responseContent).read("$[*].posts[0].title");
  8. MatcherAssert.assertThat(postTitles, Matchers.hasSize(3));
  9. MatcherAssert.assertThat(postTitles, Matchers.everyItem(Matchers.is("post1")));
  10. }
  11. </string>

Here is shown that you can easily extract repeatedly nestet items.

Tools

To test your jsonpath expression you can testers which are many on the internet.
Here you can find one of them:
https://jsonpath.curiousconcept.com/

Links

https://github.com/jayway/JsonPath
https://github.com/raphaelsolarski/jsonpath-to-test-rest-controller-example


Hamcrest collection matchers – learn by examples.

Hello.
In this post I will show how you can improve quality and clarity of your assertions using matchers in context of testing collections.

  1. import com.google.common.collect.Lists;
  2. import org.junit.Test;
  3.  
  4. import java.util.List;
  5.  
  6. import static org.hamcrest.MatcherAssert.assertThat;
  7. import static org.hamcrest.Matchers.*;
  8.  
  9. public class HamcrestCollectionsLT {
  10.  
  11. @Test
  12. public void collectionShouldHasOneGivenItem() throws Exception {
  13. List<String> list = Lists.newArrayList("Foo");
  14. assertThat(list, contains("Foo"));
  15. }
  16.  
  17. @Test
  18. public void collectionShouldHasExactlyGivenItemsInGivenOrder() throws Exception {
  19. List<String> list = Lists.newArrayList("Foo", "Bar");
  20. assertThat(list, contains("Foo", "Bar"));
  21. }
  22.  
  23. @Test
  24. public void collectionShouldHasExactlyGivenItemsInAnyOrder() throws Exception {
  25. List<String> list = Lists.newArrayList("Foo", "Bar");
  26. assertThat(list, containsInAnyOrder("Bar", "Foo"));
  27. }
  28.  
  29. @Test
  30. public void collectionShouldHasGivenItem() throws Exception {
  31. List<String> list = Lists.newArrayList("Foo", "Bar", "Element");
  32. assertThat(list, hasItem("Foo"));
  33. }
  34.  
  35. @Test
  36. public void collectionShouldNotHasGivenItem() throws Exception {
  37. List<String> list = Lists.newArrayList("Bar", "Element");
  38. assertThat(list, not(hasItem("Foo")));
  39. }
  40.  
  41. @Test
  42. public void collectionShouldBeEmpty() throws Exception {
  43. List<String> list = Lists.newArrayList();
  44. assertThat(list, empty());
  45. }
  46.  
  47. @Test
  48. public void collectionShouldNotBeEmpty() throws Exception {
  49. List<String> list = Lists.newArrayList("Foo");
  50. assertThat(list, not(empty()));
  51. }
  52.  
  53. @Test
  54. public void collectionShouldHasGivenLength() throws Exception {
  55. List<String> list = Lists.newArrayList("Foo", "Bar", "Element");
  56. assertThat(list, hasSize(3));
  57. }
  58.  
  59. @Test
  60. public void collectionShouldNotHaveGivenLength() throws Exception {
  61. List<String> list = Lists.newArrayList("Foo");
  62. assertThat(list, not(hasSize(2)));
  63. }
  64.  
  65. @Test
  66. public void everyCollectionItemShouldBeGreater() throws Exception {
  67. List<Integer> list = Lists.newArrayList(4, 5, 6);
  68. assertThat(list, everyItem(greaterThan(3)));
  69. }
  70.  
  71. @Test
  72. public void everyCollectionItemShouldBeGreaterOrEqual() throws Exception {
  73. List<Integer> list = Lists.newArrayList(4, 5, 6);
  74. assertThat(list, everyItem(greaterThanOrEqualTo(4)));
  75. }
  76.  
  77. @Test
  78. public void everyCollectionStringItemContainSubstring() throws Exception {
  79. List<String> list = Lists.newArrayList("Sub1", "Sub2", "Sub3");
  80. assertThat(list, everyItem(containsString("Sub")));
  81. }
  82.  
  83. @Test
  84. public void shouldContainAtLastGivenElementsAndEveryElementShouldContainGivenString() throws Exception {
  85. List<String> list = Lists.newArrayList("Sub1", "Sub2", "Sub3");
  86. assertThat(list, allOf(hasItems("Sub1", "Sub2"), everyItem(containsString("Sub"))));
  87. }
  88.  
  89. }

Links:
https://github.com/hamcrest/JavaHamcrest


Spring test configuration example.

Hello.
Today I want to show you simple test configuration in spring mvc project.
In example I use java based configuration(I admit that this kind of configuration I prefer).
But in future I want to present xml based configuration, too.

On the bottom of the post you find link to github repository with full project.

Class under testing:

  1. package com.raphaelsolarski.springtestconfig.controller;
  2.  
  3. import org.springframework.stereotype.Controller;
  4. import org.springframework.web.bind.annotation.RequestMapping;
  5. import org.springframework.web.bind.annotation.RequestMethod;
  6.  
  7. @Controller
  8. public class MyController {
  9.  
  10. @RequestMapping(path = "/", method = RequestMethod.GET)
  11. String home() {
  12. return "home";
  13. }
  14.  
  15. }

Test:

  1. package com.raphaelsolarski.springtestconfig.controller;
  2.  
  3. import com.raphaelsolarski.springtestconfig.config.WebConfig;
  4. import org.junit.Before;
  5. import org.junit.Test;
  6. import org.junit.runner.RunWith;
  7. import org.springframework.beans.factory.annotation.Autowired;
  8. import org.springframework.test.context.ContextConfiguration;
  9. import org.springframework.test.context.TestExecutionListeners;
  10. import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
  11. import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;
  12. import org.springframework.test.context.web.AnnotationConfigWebContextLoader;
  13. import org.springframework.test.context.web.WebAppConfiguration;
  14. import org.springframework.test.web.servlet.MockMvc;
  15. import org.springframework.test.web.servlet.result.MockMvcResultMatchers;
  16. import org.springframework.test.web.servlet.setup.MockMvcBuilders;
  17. import org.springframework.web.context.WebApplicationContext;
  18.  
  19. import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
  20.  
  21. @RunWith(SpringJUnit4ClassRunner.class)
  22. @ContextConfiguration(classes = {WebConfig.class}, loader = AnnotationConfigWebContextLoader.class)
  23. @TestExecutionListeners(listeners = {DependencyInjectionTestExecutionListener.class})
  24. @WebAppConfiguration
  25. public class MyControllerTest {
  26.  
  27. @Autowired
  28. private WebApplicationContext webApplicationContext;
  29.  
  30. private MockMvc mockMvc;
  31.  
  32. @Before
  33. public void setUp() {
  34. mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext).build();
  35. }
  36.  
  37. @Test
  38. public void getRootShouldReturnHomeView() throws Exception {
  39. mockMvc.perform(get("/")).andExpect(MockMvcResultMatchers.view().name("home"));
  40. }
  41.  
  42. }

github-icon Full code on github.com

Links:
http://docs.spring.io/spring/docs/current/spring-framework-reference/html/integration-testing.html


How to compile maven project without web.xml?

When you use servlet api beyond 3.0 version, you can use java based configuration of your webapp instead of web.xml file.
Maven by default report an error when you try to compile project that does not contain web.xml file, so you have to additionally configure maven-war-plugin in pom.xml.

  1. <plugin>
  2. <groupid>org.apache.maven.plugins</groupid>
  3. <artifactid>maven-war-plugin</artifactid>
  4. <version>2.2</version>
  5. <configuration>
  6. <failonmissingwebxml>false</failonmissingwebxml>
  7. </configuration>
  8. </plugin>