conv1d - added 1d convolution #807

jcrist1 · 2023-07-10T21:42:47Z

This PR only adds support for the conv1d operator based on Issue #219. The implementation was heavy on copy paste, and I have glossed over the index calculations. Also I was unable to verify the cuda versions, as I don't have an Nvidia card. I have an idea where we could use property based testing and PyO3 to compare with pytorch in a more rigorous way.

For GH: Resolves #219

nkoppel

This looks really good except for a few places where you forgot to update documentation. @coreylowman will need to verify this on CUDA before merging.

src/tensor_ops/conv1d/mod.rs

nkoppel · 2023-07-14T15:12:08Z

src/nn/conv1d.rs

+    }
+}
+
+/// **Requires Nightly** Performs *unbiased* 2d convolutions on 3d and 4d images.


Documentation needs an update here.

nkoppel · 2023-07-14T15:12:55Z

src/tensor_ops/conv1d/mod.rs

+    /// Applies a 2D convolution to the input tensor.
+    fn conv1d(
+        self,
+        stride: Stride,
+        padding: Padding,
+        dilation: Dilation,
+        groups: Groups,
+    ) -> Self::Convolved {
+        self.try_conv1d(stride, padding, dilation, groups).unwrap()
+    }
+
+    /// Fallibly applies a 2D convolution to the input tensor.


Documentation needs updates.

jcrist1 · 2023-07-15T17:48:14Z

@nkoppel addressed the documentation

coreylowman · 2023-07-19T12:53:32Z

src/tensor_ops/conv1d/cudnn_kernel.rs

cuDNN doesn't support 1d convolutions so we can remove this file

coreylowman · 2023-07-19T12:53:50Z

src/tensor_ops/conv1d/mod.rs

+#[cfg(feature = "cudnn")]
+mod cudnn_kernel;


noting to remove this when the cudnn stuff is removed

coreylowman · 2023-07-19T12:57:23Z

I'll test this on CUDA in the next couple days - BTW I just use https://cloud.lambdalabs.com/instances to rent GPUs to test on. If they have an A10 available its very cheap 😁

jcrist1 · 2023-07-24T20:12:38Z

After a long weekend away from the computer I'm back. I've managed to try out the compile on lambda labs, and it failed. Will dig into it a bit to see if I can figure it out.

jcrist1 · 2023-07-25T18:24:02Z

Results of

cargo test -F cuda

test result: ok. 366 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 18.52s

test result: ok. 187 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 4.07s

Edit, forgot to add nightly to features. Now I'm getting failures. Will investigate

jcrist1 · 2023-07-25T18:42:45Z

Okay it's quite broken:

running 14 tests
test nn::conv1d::tests::test_2_conv_sizes ... ok
test nn::conv1d::tests::test_3_conv_sizes ... ok
test tensor_ops::conv1d::tests::test_conv1d_s4p3k2 ... FAILED
 ** On entry to SgemmStridedBatched parameter number 10 had an illegal value
test tensor_ops::conv1d::tests::test_conv1d_grouped_forward ... FAILED
test nn::conv1d::tests::test_grouped_forward_sizes ... FAILED
test nn::conv1d::tests::test_conv_with_optimizer ... FAILED
test nn::conv1d::tests::test_forward_4d_sizes ... ok
test nn::conv1d::tests::test_forward_3d_sizes ... ok
test tensor_ops::conv1d::tests::test_conv1d_default_stride_and_padding ... FAILED
test tensor_ops::conv1d::tests::test_conv1d_stride_3_padding_4 ... FAILED
test tensor_ops::conv1d::tests::test_conv1d_dilated ... FAILED
test tensor_ops::conv1d::tests::test_conv1d_stride_2 ... FAILED
test tensor_ops::conv1d::tests::test_conv1d_padding_1 ... FAILED
test tensor_ops::conv1d::tests::test_batched_conv1d ... FAILED

failures:

---- tensor_ops::conv1d::tests::test_conv1d_s4p3k2 stdout ----
weight data [1.433521, 0.336105, 1.540528, 0.13556679, 0.03349175, -0.88206655, -0.23682937, 0.06386094, -0.3381852, 0.091517635, -0.96609956, -0.2803067, -0.028431298, 0.25178358, -0.895666, -0.043041553, 0.28816602, -0.7349546, 0.7517081, -1.4666909, 0.0781738, -0.80594444, -0.38435066, 1.0249952, -0.79651743, 0.18758032, -0.15397495, -0.32874227, 0.82089, -0.048914522]
bias data [0.44691145, 0.1279889, -0.66274095]
x data [1.4211099, 0.57097477, 0.26090318, 2.0396724, -2.2423167, -0.7642506, -0.6918944, 1.7601547, -0.979772, -0.12770805, -2.3478322, -1.1648707, -0.5200417, 0.5308337, 0.84791464, -0.03722416, -1.6566591, 0.71260184, 1.6553345, 1.6171516, 0.013870729, 0.5337378, 0.028446166, 0.72817594, 1.2460675, 0.2969481, 1.1754416, -0.57176435, 0.29306656, -0.19717735, -0.072016776, -0.24861556, 1.3089604, 0.35397184, -0.55793697]
out data [1.7531477, 0.44691145, 0.44691145, 0.10388159, 0.1279889, 0.1279889, -0.9886375, -0.66274095, -0.66274095], (Const, Const)
thread 'tensor_ops::conv1d::tests::test_conv1d_s4p3k2' panicked at 'lhs != rhs | 1.7531477 != 0.44691145', src/tensor_ops/conv1d/tests.rs:175:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- tensor_ops::conv1d::tests::test_conv1d_grouped_forward stdout ----
thread 'tensor_ops::conv1d::tests::test_conv1d_grouped_forward' panicked at 'lhs != rhs | 0.29220703 != -0.35965547', src/tensor_ops/conv1d/tests.rs:262:9

---- nn::conv1d::tests::test_grouped_forward_sizes stdout ----
thread 'nn::conv1d::tests::test_grouped_forward_sizes' panicked at 'called `Result::unwrap()` on an `Err` value: CublasError(CUBLAS_STATUS_INVALID_VALUE)', src/tensor_ops/conv1d/cuda_kernel.rs:126:22

---- nn::conv1d::tests::test_conv_with_optimizer stdout ----
thread 'nn::conv1d::tests::test_conv_with_optimizer' panicked at 'assertion failed: `(left != right)`
  left: `[[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]`,
 right: `[[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]`', src/nn/conv1d.rs:269:9

---- tensor_ops::conv1d::tests::test_conv1d_default_stride_and_padding stdout ----
thread 'tensor_ops::conv1d::tests::test_conv1d_default_stride_and_padding' panicked at 'lhs != rhs | -0.41916567 != 0.2249364', src/tensor_ops/conv1d/tests.rs:23:5

---- tensor_ops::conv1d::tests::test_conv1d_stride_3_padding_4 stdout ----
thread 'tensor_ops::conv1d::tests::test_conv1d_stride_3_padding_4' panicked at 'lhs != rhs | 381905320000000000000 != -0.31665158', src/tensor_ops/conv1d/tests.rs:136:5

---- tensor_ops::conv1d::tests::test_conv1d_dilated stdout ----
thread 'tensor_ops::conv1d::tests::test_conv1d_dilated' panicked at 'lhs != rhs | 0 != 1', src/tensor_ops/conv1d/tests.rs:225:5

---- tensor_ops::conv1d::tests::test_conv1d_stride_2 stdout ----
thread 'tensor_ops::conv1d::tests::test_conv1d_stride_2' panicked at 'lhs != rhs | -0.16069691 != -0.03002486', src/tensor_ops/conv1d/tests.rs:55:5

---- tensor_ops::conv1d::tests::test_conv1d_padding_1 stdout ----
thread 'tensor_ops::conv1d::tests::test_conv1d_padding_1' panicked at 'lhs != rhs | 0.13320218 != 0.06176047', src/tensor_ops/conv1d/tests.rs:90:5

---- tensor_ops::conv1d::tests::test_batched_conv1d stdout ----
thread 'tensor_ops::conv1d::tests::test_batched_conv1d' panicked at 'lhs != rhs | 0.23777176 != 0.50760746', src/tensor_ops/conv1d/tests.rs:206:9


failures:
    nn::conv1d::tests::test_conv_with_optimizer
    nn::conv1d::tests::test_grouped_forward_sizes
    tensor_ops::conv1d::tests::test_batched_conv1d
    tensor_ops::conv1d::tests::test_conv1d_default_stride_and_padding
    tensor_ops::conv1d::tests::test_conv1d_dilated
    tensor_ops::conv1d::tests::test_conv1d_grouped_forward
    tensor_ops::conv1d::tests::test_conv1d_padding_1
    tensor_ops::conv1d::tests::test_conv1d_s4p3k2
    tensor_ops::conv1d::tests::test_conv1d_stride_2
    tensor_ops::conv1d::tests::test_conv1d_stride_3_padding_4

coreylowman · 2023-07-31T13:23:13Z

Thanks for testing - and yeah debugging cuda code is a pain 😅

jcrist1 · 2023-08-12T09:14:33Z

Sorry this fell by the wayside, had a lot going on the last two weeks. Will try to dig into this now

coreylowman · 2023-08-14T18:07:20Z

No worries at all, real life happens a lot in the open source community 😁

coreylowman · 2023-09-07T17:26:07Z

@jcrist1 I'm going to merge this into a conv1d branch so we can collab on it, I'll start pushing to that branch. Trying to get this implemented so I can test out a whisper implementation

* conv1d - added 1d convolution (#807) * conv1d - added 1d convolution * conv1d - quick port of cuda code from conv2d -> conv1d * conv1d - extra nightly to get through clippy * conv1d - remove duplicates * conv1d - clean up comments * conv1d - remove cudnn * fixes --------- Co-authored-by: Corey Lowman <clowman1993@gmail.com> * Simplify conv1d tests * Fixing cpu kernel * Updating cuda kernels * Fixing bugs * Reverting debugging changes * Fixing grouped conv1d cuda kernels --------- Co-authored-by: jcrist1 <jan.cristina@gmail.com>

jcrist1 · 2023-09-08T07:17:29Z

@coreylowman thanks, I'm mostly throwing spaghetti at the wall trying to figure out why the buffers don't work. There's a lot of garbage now especially in the cuda code with println debugging

jcrist1 added 4 commits July 10, 2023 23:31

conv1d - added 1d convolution

56cc462

Merge remote-tracking branch 'origin/main' into conv1d

ca3880e

conv1d - quick port of cuda code from conv2d -> conv1d

b31ad78

conv1d - extra nightly to get through clippy

0224d40

jcrist1 changed the title ~~conv1d - added 1d convolution~~ [WIP] conv1d - added 1d convolution Jul 11, 2023

conv1d - remove duplicates

019dbbf

jcrist1 marked this pull request as ready for review July 13, 2023 14:26

jcrist1 changed the title ~~[WIP] conv1d - added 1d convolution~~ conv1d - added 1d convolution Jul 13, 2023

nkoppel reviewed Jul 14, 2023

View reviewed changes

conv1d - clean up comments

7bc71fb

jcrist1 requested a review from nkoppel July 15, 2023 17:48

coreylowman reviewed Jul 19, 2023

View reviewed changes

src/tensor_ops/conv1d/cudnn_kernel.rs Outdated

Copy link

Owner

coreylowman Jul 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cuDNN doesn't support 1d convolutions so we can remove this file

coreylowman reviewed Jul 19, 2023

View reviewed changes

Merge branch 'main' into conv1d

a6b30da

jcrist1 added 3 commits July 25, 2023 21:17

Merge remote-tracking branch 'origin/main' into conv1d

5dac4b4

Merge branch 'conv1d' of github.com:jcrist1/dfdx into conv1d

4fc4a5b

conv1d - remove cudnn

5eb1609

jcrist1 added 5 commits August 14, 2023 21:31

fixes

d181489

conv1 - some fixes

325d44e

Merge remote-tracking branch 'origin/main' into conv1d

145a25b

trying something that doesn't work

932c80a

debug

b6c63ca

jcrist1 added 11 commits August 25, 2023 21:29

more debug

e60f437

more debug

8d56463

more debug

f47d842

more debug

d9b506e

more debug

5011f3d

more debug

ad222b6

more debug

79ed19c

more debug

11e0a50

more debug

57b836b

more debug

7f2d823

more debug

cf0109a

jcrist1 force-pushed the conv1d branch from cf62769 to cf0109a Compare August 31, 2023 20:07

conv1d - more debugging

9c319be

coreylowman changed the base branch from main to conv1d September 7, 2023 17:25

coreylowman merged commit cab64bd into coreylowman:conv1d Sep 7, 2023
2 of 4 checks passed

jcrist1 deleted the conv1d branch September 8, 2023 07:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

conv1d - added 1d convolution #807

conv1d - added 1d convolution #807

jcrist1 commented Jul 10, 2023 •

edited

Loading

nkoppel left a comment

nkoppel Jul 14, 2023

nkoppel Jul 14, 2023

jcrist1 commented Jul 15, 2023

coreylowman Jul 19, 2023

coreylowman Jul 19, 2023

coreylowman commented Jul 19, 2023

jcrist1 commented Jul 24, 2023

jcrist1 commented Jul 25, 2023 •

edited

Loading

jcrist1 commented Jul 25, 2023 •

edited

Loading

coreylowman commented Jul 31, 2023

jcrist1 commented Aug 12, 2023

coreylowman commented Aug 14, 2023

coreylowman commented Sep 7, 2023

jcrist1 commented Sep 8, 2023

		#[cfg(feature = "cudnn")]
		mod cudnn_kernel;

conv1d - added 1d convolution #807

conv1d - added 1d convolution #807

Conversation

jcrist1 commented Jul 10, 2023 • edited Loading

nkoppel left a comment

Choose a reason for hiding this comment

nkoppel Jul 14, 2023

Choose a reason for hiding this comment

nkoppel Jul 14, 2023

Choose a reason for hiding this comment

jcrist1 commented Jul 15, 2023

coreylowman Jul 19, 2023

Choose a reason for hiding this comment

coreylowman Jul 19, 2023

Choose a reason for hiding this comment

coreylowman commented Jul 19, 2023

jcrist1 commented Jul 24, 2023

jcrist1 commented Jul 25, 2023 • edited Loading

jcrist1 commented Jul 25, 2023 • edited Loading

coreylowman commented Jul 31, 2023

jcrist1 commented Aug 12, 2023

coreylowman commented Aug 14, 2023

coreylowman commented Sep 7, 2023

jcrist1 commented Sep 8, 2023

jcrist1 commented Jul 10, 2023 •

edited

Loading

jcrist1 commented Jul 25, 2023 •

edited

Loading

jcrist1 commented Jul 25, 2023 •

edited

Loading