Skip to content

fix(bookshelf-standard): don't crash startup on index lag #1266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
saiflayouni wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(bookshelf-standard): don't crash startup on index lag #1266

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions ...d/2-structured-data/src/main/java/com/example/getstarted/util/DatastoreSessionFilter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

package com.example.getstarted.util;

import com.google.appengine.api.datastore.DatastoreNeedIndexException;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
Expand All @@ -35,6 +36,8 @@
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
Expand All @@ -53,6 +56,8 @@
// [START init]
public class DatastoreSessionFilter implements Filter {

private static final Logger logger = Logger.getLogger(DatastoreSessionFilter.class.getName());

private static DatastoreService datastore;
private static final DateTimeFormatter DTF = DateTimeFormat.forPattern("yyyyMMddHHmmssSSS");
private static final String SESSION_KIND = "SessionVariable";
Expand All @@ -62,14 +67,21 @@ public void init(FilterConfig config) throws ServletException {
// initialize local copy of datastore session variables

datastore = DatastoreServiceFactory.getDatastoreService();
// Delete all sessions unmodified for over two days
DateTime dt = DateTime.now(DateTimeZone.UTC);
Query query = new Query(SESSION_KIND).setFilter(new FilterPredicate(
"lastModified", FilterOperator.LESS_THAN_OR_EQUAL, dt.minusDays(2).toString(DTF)));
Iterator<Entity> results = datastore.prepare(query).asIterator();
while (results.hasNext()) {
Entity stateEntity = results.next();
datastore.delete(stateEntity.getKey());
// Delete all sessions unmodified for over two days. Best-effort: on a fresh
// deployment Datastore has no index yet for a property it has never seen
// written, so this throws DatastoreNeedIndexException until the first session
// is written. Don't let that take down the whole filter (and app) at startup.
try {
DateTime dt = DateTime.now(DateTimeZone.UTC);
Query query = new Query(SESSION_KIND).setFilter(new FilterPredicate(
"lastModified", FilterOperator.LESS_THAN_OR_EQUAL, dt.minusDays(2).toString(DTF)));
Iterator<Entity> results = datastore.prepare(query).asIterator();
while (results.hasNext()) {
Entity stateEntity = results.next();
datastore.delete(stateEntity.getKey());
}
Comment on lines +78 to +82

@gemini-code-assist gemini-code-assist Bot Oct 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Performing Datastore delete operations one-by-one in a loop is highly inefficient as it triggers a separate RPC for each entity. Since this cleanup runs synchronously during filter initialization (init()), a large number of stale sessions could cause significant startup delays or timeouts. Batching the deletes into a single datastore.delete(keysToDelete) call is much more efficient and safer.

      Iterable<Entity> results = datastore.prepare(query).asIterable();
      java.util.List<com.google.appengine.api.datastore.Key> keysToDelete = new java.util.ArrayList<>();
      for (Entity stateEntity : results) {
        keysToDelete.add(stateEntity.getKey());
      }
      if (!keysToDelete.isEmpty()) {
        datastore.delete(keysToDelete);
      }

} catch (DatastoreNeedIndexException e) {
logger.log(Level.WARNING, "Skipping stale-session cleanup; index not ready yet", e);
}
}
// [END init]
Expand Down
28 changes: 20 additions & 8 deletions ...ndard/3-binary-data/src/main/java/com/example/getstarted/util/DatastoreSessionFilter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

package com.example.getstarted.util;

import com.google.appengine.api.datastore.DatastoreNeedIndexException;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
Expand All @@ -35,6 +36,8 @@
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
Expand All @@ -53,6 +56,8 @@
// [START init]
public class DatastoreSessionFilter implements Filter {

private static final Logger logger = Logger.getLogger(DatastoreSessionFilter.class.getName());

private static DatastoreService datastore;
private static final DateTimeFormatter DTF = DateTimeFormat.forPattern("yyyyMMddHHmmssSSS");
private static final String SESSION_KIND = "SessionVariable";
Expand All @@ -62,14 +67,21 @@ public void init(FilterConfig config) throws ServletException {
// initialize local copy of datastore session variables

datastore = DatastoreServiceFactory.getDatastoreService();
// Delete all sessions unmodified for over two days
DateTime dt = DateTime.now(DateTimeZone.UTC);
Query query = new Query(SESSION_KIND).setFilter(new FilterPredicate(
"lastModified", FilterOperator.LESS_THAN_OR_EQUAL, dt.minusDays(2).toString(DTF)));
Iterator<Entity> results = datastore.prepare(query).asIterator();
while (results.hasNext()) {
Entity stateEntity = results.next();
datastore.delete(stateEntity.getKey());
// Delete all sessions unmodified for over two days. Best-effort: on a fresh
// deployment Datastore has no index yet for a property it has never seen
// written, so this throws DatastoreNeedIndexException until the first session
// is written. Don't let that take down the whole filter (and app) at startup.
try {
DateTime dt = DateTime.now(DateTimeZone.UTC);
Query query = new Query(SESSION_KIND).setFilter(new FilterPredicate(
"lastModified", FilterOperator.LESS_THAN_OR_EQUAL, dt.minusDays(2).toString(DTF)));
Iterator<Entity> results = datastore.prepare(query).asIterator();
while (results.hasNext()) {
Entity stateEntity = results.next();
datastore.delete(stateEntity.getKey());
}
Comment on lines +78 to +82

@gemini-code-assist gemini-code-assist Bot Oct 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Performing Datastore delete operations one-by-one in a loop is highly inefficient as it triggers a separate RPC for each entity. Since this cleanup runs synchronously during filter initialization (init()), a large number of stale sessions could cause significant startup delays or timeouts. Batching the deletes into a single datastore.delete(keysToDelete) call is much more efficient and safer.

      Iterable<Entity> results = datastore.prepare(query).asIterable();
      java.util.List<com.google.appengine.api.datastore.Key> keysToDelete = new java.util.ArrayList<>();
      for (Entity stateEntity : results) {
        keysToDelete.add(stateEntity.getKey());
      }
      if (!keysToDelete.isEmpty()) {
        datastore.delete(keysToDelete);
      }

} catch (DatastoreNeedIndexException e) {
logger.log(Level.WARNING, "Skipping stale-session cleanup; index not ready yet", e);
}
}
// [END init]
Expand Down
28 changes: 20 additions & 8 deletions ...elf-standard/4-auth/src/main/java/com/example/getstarted/util/DatastoreSessionFilter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

package com.example.getstarted.util;

import com.google.appengine.api.datastore.DatastoreNeedIndexException;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
Expand All @@ -35,6 +36,8 @@
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
Expand All @@ -53,6 +56,8 @@
// [START init]
public class DatastoreSessionFilter implements Filter {

private static final Logger logger = Logger.getLogger(DatastoreSessionFilter.class.getName());

private static DatastoreService datastore;
private static final DateTimeFormatter DTF = DateTimeFormat.forPattern("yyyyMMddHHmmssSSS");
private static final String SESSION_KIND = "SessionVariable";
Expand All @@ -62,14 +67,21 @@ public void init(FilterConfig config) throws ServletException {
// initialize local copy of datastore session variables

datastore = DatastoreServiceFactory.getDatastoreService();
// Delete all sessions unmodified for over two days
DateTime dt = DateTime.now(DateTimeZone.UTC);
Query query = new Query(SESSION_KIND).setFilter(new FilterPredicate(
"lastModified", FilterOperator.LESS_THAN_OR_EQUAL, dt.minusDays(2).toString(DTF)));
Iterator<Entity> results = datastore.prepare(query).asIterator();
while (results.hasNext()) {
Entity stateEntity = results.next();
datastore.delete(stateEntity.getKey());
// Delete all sessions unmodified for over two days. Best-effort: on a fresh
// deployment Datastore has no index yet for a property it has never seen
// written, so this throws DatastoreNeedIndexException until the first session
// is written. Don't let that take down the whole filter (and app) at startup.
try {
DateTime dt = DateTime.now(DateTimeZone.UTC);
Query query = new Query(SESSION_KIND).setFilter(new FilterPredicate(
"lastModified", FilterOperator.LESS_THAN_OR_EQUAL, dt.minusDays(2).toString(DTF)));
Iterator<Entity> results = datastore.prepare(query).asIterator();
while (results.hasNext()) {
Entity stateEntity = results.next();
datastore.delete(stateEntity.getKey());
}
Comment on lines +78 to +82

@gemini-code-assist gemini-code-assist Bot Oct 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Performing Datastore delete operations one-by-one in a loop is highly inefficient as it triggers a separate RPC for each entity. Since this cleanup runs synchronously during filter initialization (init()), a large number of stale sessions could cause significant startup delays or timeouts. Batching the deletes into a single datastore.delete(keysToDelete) call is much more efficient and safer.

      Iterable<Entity> results = datastore.prepare(query).asIterable();
      java.util.List<com.google.appengine.api.datastore.Key> keysToDelete = new java.util.ArrayList<>();
      for (Entity stateEntity : results) {
        keysToDelete.add(stateEntity.getKey());
      }
      if (!keysToDelete.isEmpty()) {
        datastore.delete(keysToDelete);
      }

} catch (DatastoreNeedIndexException e) {
logger.log(Level.WARNING, "Skipping stale-session cleanup; index not ready yet", e);
}
}
// [END init]
Expand Down
28 changes: 20 additions & 8 deletions ...-standard/5-logging/src/main/java/com/example/getstarted/util/DatastoreSessionFilter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

package com.example.getstarted.util;

import com.google.appengine.api.datastore.DatastoreNeedIndexException;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
Expand All @@ -35,6 +36,8 @@
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
Expand All @@ -53,6 +56,8 @@
// [START init]
public class DatastoreSessionFilter implements Filter {

private static final Logger logger = Logger.getLogger(DatastoreSessionFilter.class.getName());

private static DatastoreService datastore;
private static final DateTimeFormatter DTF = DateTimeFormat.forPattern("yyyyMMddHHmmssSSS");
private static final String SESSION_KIND = "SessionVariable";
Expand All @@ -62,14 +67,21 @@ public void init(FilterConfig config) throws ServletException {
// initialize local copy of datastore session variables

datastore = DatastoreServiceFactory.getDatastoreService();
// Delete all sessions unmodified for over two days
DateTime dt = DateTime.now(DateTimeZone.UTC);
Query query = new Query(SESSION_KIND).setFilter(new FilterPredicate(
"lastModified", FilterOperator.LESS_THAN_OR_EQUAL, dt.minusDays(2).toString(DTF)));
Iterator<Entity> results = datastore.prepare(query).asIterator();
while (results.hasNext()) {
Entity stateEntity = results.next();
datastore.delete(stateEntity.getKey());
// Delete all sessions unmodified for over two days. Best-effort: on a fresh
// deployment Datastore has no index yet for a property it has never seen
// written, so this throws DatastoreNeedIndexException until the first session
// is written. Don't let that take down the whole filter (and app) at startup.
try {
DateTime dt = DateTime.now(DateTimeZone.UTC);
Query query = new Query(SESSION_KIND).setFilter(new FilterPredicate(
"lastModified", FilterOperator.LESS_THAN_OR_EQUAL, dt.minusDays(2).toString(DTF)));
Iterator<Entity> results = datastore.prepare(query).asIterator();
while (results.hasNext()) {
Entity stateEntity = results.next();
datastore.delete(stateEntity.getKey());
}
Comment on lines +78 to +82

@gemini-code-assist gemini-code-assist Bot Oct 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Performing Datastore delete operations one-by-one in a loop is highly inefficient as it triggers a separate RPC for each entity. Since this cleanup runs synchronously during filter initialization (init()), a large number of stale sessions could cause significant startup delays or timeouts. Batching the deletes into a single datastore.delete(keysToDelete) call is much more efficient and safer.

      Iterable<Entity> results = datastore.prepare(query).asIterable();
      java.util.List<com.google.appengine.api.datastore.Key> keysToDelete = new java.util.ArrayList<>();
      for (Entity stateEntity : results) {
        keysToDelete.add(stateEntity.getKey());
      }
      if (!keysToDelete.isEmpty()) {
        datastore.delete(keysToDelete);
      }

} catch (DatastoreNeedIndexException e) {
logger.log(Level.WARNING, "Skipping stale-session cleanup; index not ready yet", e);
}
}
// [END init]
Expand Down
Loading