Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: txn_coord_sender.go:1013: transaction unexpectedly committed, ba: Put [/Table/109/1/×/×/×/×,/Min), EndTxn(parallel commit) [/Table/109/1/×/×/×/×], [txn: 74d3975b]. txn: "sql txn" meta=... #124337

Closed
cockroach-sentry opened this issue May 17, 2024 · 1 comment
Closed

Sentry: txn_coord_sender.go:1013: transaction unexpectedly committed, ba: Put [/Table/109/1/×/×/×/×,/Min), EndTxn(parallel commit) [/Table/109/1/×/×/×/×], [txn: 74d3975b]. txn: "sql txn" meta=... #124337

cockroach-sentry opened this issue May 17, 2024 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented May 17, 2024 •
edited by cockroach-jira-scripts

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5363715484/?referrer=webhooks_plugin

Panic Message:

txn_coord_sender.go:1013: transaction unexpectedly committed, ba: Put [/Table/109/1/×/×/×/×,/Min), EndTxn(parallel commit) [/Table/109/1/×/×/×/×], [txn: 74d3975b]. txn: "sql txn" meta={id=74d3975b key=/Table/109/1/×/×/×/× pri=0.03113798 epo=0 ts=1715935150.977260095,1 min=1715935140.728199291,0 seq=246} lock=true stat=COMMITTED rts=1715935150.977260095,1 wto=false gul=1715935141.228199291,0 int=245 ifw=245: TransactionStatusError: already committed (REASON_TXN_COMMITTED)
(1)
Wraps: (2) assertion failure
Wraps: (3) issue: https://github.com/cockroachdb/cockroach/issues/103817
  | detail: you have encountered a known bug in CockroachDB, please consider reporting on the Github issue or reach out via Support.
Wraps: (4)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1013
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:977
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:537
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1036
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1060
  | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:860
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:675
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).CommitInBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:723
  | github.com/cockroachdb/cockroach/pkg/sql.(*tableWriterBase).finalize
  | 	github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go:230
  | github.com/cockroachdb/cockroach/pkg/sql.(*upsertNode).BatchedNext
  | 	github.com/cockroachdb/cockroach/pkg/sql/upsert.go:125
  | github.com/cockroachdb/cockroach/pkg/sql.(*rowCountNode).startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:173
  | github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:519
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:299
  | github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
  | github.com/cockroachdb/cockroach/pkg/sql.walkPlan
  | 	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
  | github.com/cockroachdb/cockroach/pkg/sql.startExec
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan.go:522
  | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:175
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:183
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:229
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:228
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:120
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:119
  | github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:725
  | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:580
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:313
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:902
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1895
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1627
  | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
  | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1630
Wraps: (5) transaction unexpectedly committed, ba: Put [/Table/109/1/×/×/×/×,/Min), EndTxn(parallel commit) [/Table/109/1/×/×/×/×], [txn: 74d3975b]. txn: "sql txn" meta={id=74d3975b key=/Table/109/1/×/×/×/× pri=0.03113798 epo=0 ts=1715935150.977260095,1 min=1715935140.728199291,0 seq=246} lock=true stat=COMMITTED rts=1715935150.977260095,1 wto=false gul=1715935141.228199291,0 int=245 ifw=245
Wraps: (6) TransactionStatusError: already committed (REASON_TXN_COMMITTED)
Error types: (1) *colexecerror.notInternalError (2) *assert.withAssertionFailure (3) *issuelink.withIssueLink (4) *errbase.opaqueWrapper (5) *errutil.withPrefix (6) *kvpb.TransactionStatusError
-- report composition:
*kvpb.TransactionStatusError
*errutil.withPrefix: transaction unexpectedly committed, ba: Put [/Table/109/1/×/×/×/×,/Min), EndTxn(parallel commit) [/Table/109/1/×/×/×/×], [txn: 74d3975b]. txn: "sql txn" meta={id=74d3975b key=/Table/109/1/×/×/×/× pri=0.03113798 epo=0 ts=1715935150.977260095,1 min=1715935140.728199291,0 seq=246} lock=true stat=COMMITTED rts=1715935150.977260095,1 wto=false gul=1715935141.228199291,0 int=245 ifw=245
txn_coord_sender.go:1013: *withstack.withStack (top exception)
*issuelink.withIssueLink: https://github.com/cockroachdb/cockroach/issues/103817
*assert.withAssertionFailure
*colexecerror.notInternalError
Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/distsql_running.go

Lines 1629 to 1631 in 93a6798

)
}()

cockroach/pkg/sql/distsql_running.go

Lines 1626 to 1628 in 93a6798

defer cleanup()
dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv, finishedSetupFn,

cockroach/pkg/sql/distsql_running.go

Lines 1894 to 1896 in 93a6798

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, finishedSetupFn)
}

cockroach/pkg/sql/distsql_running.go

Lines 901 to 903 in 93a6798

noWait := planCtx.getPortalPauseInfo() != nil
flow.Run(ctx, noWait)
}

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 312 to 314 in 93a6798

// as the root, so we run this flow with the default implementation.
f.FlowBase.Run(ctx, noWait)
return

cockroach/pkg/sql/flowinfra/flow.go

Lines 579 to 581 in 93a6798

f.headProcStarted = true
headProc.Run(ctx, headOutput)
}

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 724 to 726 in 93a6798

}
pb.self.Start(ctx)
Run(pb.ctx, pb.self, output)

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 118 to 120 in 93a6798

ctx = f.StartInternal(ctx, "flow coordinator" /* name */)
if err := colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Start(ctx)

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 93a6798

}()
operation()
return retErr

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 119 to 121 in 93a6798

if err := colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Start(ctx)
}); err != nil {

cockroach/pkg/sql/colexec/materializer.go

Lines 227 to 229 in 93a6798

// reached), so we need to wrap it with a catcher.
if err := colexecerror.CatchVectorizedRuntimeError(func() {
m.input.Init(ctx)

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 93a6798

}()
operation()
return retErr

cockroach/pkg/sql/colexec/materializer.go

Lines 228 to 230 in 93a6798

if err := colexecerror.CatchVectorizedRuntimeError(func() {
m.input.Init(ctx)
}); err != nil {

cockroach/pkg/sql/colexec/columnarizer.go

Lines 182 to 184 in 93a6798

ctx = c.StartInternal(ctx, "columnarizer" /* name */)
c.input.Start(ctx)
if execStatsHijacker, ok := c.input.(execinfra.ExecStatsForTraceHijacker); ok {

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 174 to 176 in 93a6798

// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)

cockroach/pkg/sql/plan.go

Lines 521 to 523 in 93a6798

}
return walkPlan(params.ctx, plan, o)
}

cockroach/pkg/sql/walk.go

Lines 42 to 44 in 93a6798

v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err

cockroach/pkg/sql/walk.go

Lines 78 to 80 in 93a6798

}
v.visitInternal(plan, name)
return plan

cockroach/pkg/sql/walk.go

Lines 298 to 300 in 93a6798

}
}

cockroach/pkg/sql/walk.go

Lines 111 to 113 in 93a6798

}
v.err = v.observer.leaveNode(name, plan)
}()

cockroach/pkg/sql/plan.go

Lines 518 to 520 in 93a6798

}
return n.startExec(params)
},

cockroach/pkg/sql/plan_batch.go

Lines 172 to 174 in 93a6798

for {
if next, err := r.source.BatchedNext(params); !next {
return err

cockroach/pkg/sql/upsert.go

Lines 124 to 126 in 93a6798

n.run.tw.setRowsWrittenLimit(params.extendedEvalCtx.SessionData())
if err := n.run.tw.finalize(params.ctx); err != nil {
return false, err

cockroach/pkg/sql/tablewriter.go

Lines 229 to 231 in 93a6798

// coordinator.
err = tb.txn.CommitInBatch(ctx, tb.b)
} else {

cockroach/pkg/kv/txn.go

Lines 722 to 724 in 93a6798

b.initResult(1 /* calls */, 0, b.raw, nil)
return txn.Run(ctx, b)
}

cockroach/pkg/kv/txn.go

Lines 674 to 676 in 93a6798

}
return sendAndFill(ctx, txn.Send, b)
}

cockroach/pkg/kv/db.go

Lines 859 to 861 in 93a6798

ba.AdmissionHeader = b.AdmissionHeader
b.response, b.pErr = send(ctx, ba)
b.fillResults(ctx)

cockroach/pkg/kv/txn.go

Lines 1059 to 1061 in 93a6798

txn.mu.Unlock()
br, pErr := txn.db.sendUsingSender(ctx, ba, sender)
if pErr == nil {

cockroach/pkg/kv/db.go

Lines 1035 to 1037 in 93a6798

br, pErr := sender.Send(ctx, ba)
if pErr != nil {

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 536 to 538 in 93a6798

pErr = tc.updateStateLocked(ctx, ba, br, pErr)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 976 to 978 in 93a6798

if errTxn := pErr.GetTxn(); errTxn != nil {
if err := sanityCheckErrWithTxn(ctx, pErr, ba, &tc.testingKnobs); err != nil {
return kvpb.NewError(err)

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 1012 to 1014 in 93a6798

// are committed.
err := errors.Wrapf(pErrWithTxn.GoError(),
"transaction unexpectedly committed, ba: %s. txn: %s", 

pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1630
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3 at line 1627
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1895
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 902
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 313
pkg/sql/flowinfra/flow.go in pkg/sql/flowinfra.(*FlowBase).Run at line 580
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run at line 725
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Start at line 119
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Start.func1 at line 120
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Start at line 228
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Start.func1 at line 229
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Init at line 183
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 175
pkg/sql/plan.go in pkg/sql.startExec at line 522
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 299
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 519
pkg/sql/plan_batch.go in pkg/sql.(*rowCountNode).startExec at line 173
pkg/sql/upsert.go in pkg/sql.(*upsertNode).BatchedNext at line 125
pkg/sql/tablewriter.go in pkg/sql.(*tableWriterBase).finalize at line 230
pkg/kv/txn.go in pkg/kv.(*Txn).CommitInBatch at line 723
pkg/kv/txn.go in pkg/kv.(*Txn).Run at line 675
pkg/kv/db.go in pkg/kv.sendAndFill at line 860
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1060
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1036
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 537
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).updateStateLocked at line 977
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.sanityCheckErrWithTxn at line 1013

Tags

Tag Value
Command server
Environment v23.1.20
Go Version go1.19.13
Platform linux arm64
Distribution CCL
Cockroach Release v23.1.20
Cockroach SHA 93a6798
# of CPUs 32
# of Goroutines 4507

Jira issue: CRDB-38835
@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels May 17, 2024
@yuzefovich
Copy link
Member

dup of #123976

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-sentry