Hibernate second-level cache stops all database operation
The customer reported today that one of our services stopped working, the new coming data can not be saved into the database.
After checking the logs, there are a lot of “Hazelcast instance is not active” exceptions, which stopped the database CRUD operations.
I assumed that the memory of the service exhausted already. The Grafana has proved my assumption. The pre-defined memory limit is 8 GB and the service was still running hard.
In this service, the second-level cache of the entity is enabled.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
@Entity @org.hibernate.annotations.Cache(usage = CacheConcurrencyStrategy.READ_WRITE) public class Foo { @Id @GeneratedValue(strategy = GenerationType.AUTO) @Column(name = "ID") private long id; @Column(name = "NAME") private String name; // getters and setters } |
Hazelcast memory is used in this service.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
jpa: database-platform: org.hibernate.dialect.MySQL5InnoDBDialect database: MYSQL show-sql: true properties: hibernate.id.new_generator_mappings: true hibernate.connection.provider_disables_autocommit: true hibernate.cache.<strong>use_second_level_cache</strong>: true hibernate.cache.use_query_cache: false hibernate.generate_statistics: true hibernate.cache.region.factory_class: com.hazelcast.hibernate.HazelcastCacheRegionFactory hibernate.cache.hazelcast.instance_name: application-name hibernate.cache.use_minimal_puts: true hibernate.cache.hazelcast.use_lite_member: true |
1. What Is a Second-Level Cache?
As most other fully-equipped ORM frameworks, Hibernate has the concept of first-level cache. It is a session scoped cache which ensures that each entity instance is loaded only once in the persistent context.
Once the session is closed, first-level cache is terminated as well. This is actually desirable, as it allows for concurrent sessions to work with entity instances in isolation from each other.
On the other hand, second-level cache is SessionFactory-scoped, meaning it is shared by all sessions created with the same session factory. When an entity instance is looked up by its id (either by application logic or by Hibernate internally, e.g. when it loads associations to that entity from other entities), and if second-level caching is enabled for that entity, the following happens:
- If an instance is already present in the first-level cache, it is returned from there
- If an instance is not found in the first-level cache, and the corresponding instance state is cached in the second-level cache, then the data is fetched from there and an instance is assembled and returned
- Otherwise, the necessary data are loaded from the database and an instance is assembled and returned
Once the instance is stored in the persistence context (first-level cache), it is returned from there in all subsequent calls within the same session until the session is closed or the instance is manually evicted from the persistence context. Also, the loaded instance state is stored in L2 cache if it was not there already.
But in our service, there is not much logic for loading/querying data from the database, the main processing is save the new coming data into the database and send the data out via Message Queue.
So we decide to disable the Hibernate Second-level Cache in this service. So that when the memory of service is nearly exhausting, it can still save new data.
1 |
hibernate.cache.use_second_level_cache=<span class="code-keyword">false</span> |