CN103994728A - Object detector and robot system - Google Patents

Object detector and robot system Download PDF

Info

Publication number
CN103994728A
CN103994728A CN201310740927.4A CN201310740927A CN103994728A CN 103994728 A CN103994728 A CN 103994728A CN 201310740927 A CN201310740927 A CN 201310740927A CN 103994728 A CN103994728 A CN 103994728A
Authority
CN
China
Prior art keywords
light source
lens
detection device
mentioned
workpiece
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201310740927.4A
Other languages
Chinese (zh)
Inventor
一丸勇二
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yaskawa Electric Corp
Original Assignee
Yaskawa Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yaskawa Electric Corp filed Critical Yaskawa Electric Corp
Publication of CN103994728A publication Critical patent/CN103994728A/en
Pending legal-status Critical Current

Links

Abstract

The invention provides a three dimensional shape object detector and robot system capable of carrying out high precision shooting and high precision detection; a sensor unit (300) comprises the following structures: a laser light source (321) emitting laser slit light (L); a rotary mirror (322) and a motor (323) enabling the laser slit light (L) emitted by the laser light source (321) to move to a projection portion of a work piece (W) in a regulated direction (A); a camera (310) used for continuously shooting, in an expected frame rate, a profile of the work piece (W) containing the moving projection portion when the motor makes the rotary mirror rotating so as to enable the laser slit light (L) to move on the projection portion of the work piece (W); lens units (350, 360) arranged on the laser light source (321) and the camera (310); a feature adjusting portion (339) used for adjusting optical features of the lens units (350, 360) when the camera (310) is continuously shooting.

Description

Article detection device and robot system
Technical field
Embodiments of the present invention relate to article detection device and robot system.
Background technology
In patent documentation 1, recorded the technology of carrying out object detection by light cross-section method.In light cross-section method, by the light irradiating from light source is moved to direction of scanning to the projection position of project objects, with desired interval, the outward appearance of the object at the projection position that comprises this and move is made a video recording continuously, thus the three-dimensional outer shape of inspected object.
Prior art document
Patent documentation
Patent documentation 1: TOHKEMY 2009-250844 communique
Summary of the invention
In light cross-section method as above, thereby the high-precision shooting three-dimensional outer shape of inspected object is accurately carried out in expectation.
The present invention makes in view of such problem, and its object is, provides a kind of and can carry out high-precision shooting and article detection device and the robot system of the three-dimensional outer shape of inspected object accurately.
Means for technical solution problem
In order to address the above problem, according to a viewpoint of the present invention, provide a kind of article detection device, the object of detected object is detected, it is characterized in that having: light source, described light source irradiates light; Scanner section, described scanner section makes the described light irradiating from described light source move to the direction of scanning of regulation to the projection position of described project objects; Video camera, when described scanner section moves the described Projection Division displacement of described light on described object, described video camera is made a video recording with the interval of expectation continuously to the outward appearance of the described object at the projection position that comprises described movement; Lens unit, described lens unit is arranged at described light source or described video camera; And characteristic adjustment part, the optical characteristics of described lens unit can be adjusted in described characteristic adjustment part when described video camera is made a video recording continuously.
Invention effect
Therefore according to the present invention, can carry out high-precision shooting, the three-dimensional outer shape of inspected object accurately.
Accompanying drawing explanation
Fig. 1 means the system pie graph of the example that the integral body of the robot system of an embodiment forms.
Fig. 2 is the side view with the state representation of part perspective by an example of the formation of sensor unit.
Fig. 3 is the side view with the state representation of part perspective by an example of the structure of sensor unit.
Fig. 4 is for the key diagram of an example of the structure of sensor unit is described.
Fig. 5 is for the key diagram of an example of the formation of sensor unit is described.
Fig. 6 means from the key diagram of an example of the shooting frame of video camera output.
Fig. 7 is the schematic diagram of an example that schematically shows the formation of lens unit.
Fig. 8 means the block diagram of an example of the functional formation of sensor controller.
Fig. 9 is for the key diagram of the focal position of liquid lens is described.
Figure 10 means the key diagram of an example of a plurality of shooting frames of exporting from video camera when the N time scanning and the range image generating by range image generating unit.
Figure 11 means the key diagram of an example of a plurality of shooting frames of exporting from video camera when the scanning of the N+1 time and the range image generating by range image generating unit.
Figure 12 means the key diagram of an example of the composograph generating by composograph generating unit.
description of reference numerals
1 robot system
20 accumulators (container)
100 robots
200 robot controllers (controller)
300 sensor units (article detection device)
310 video cameras
320 LASER Light Source unit
321 LASER Light Source (light source)
322 rotating mirrors
323 motors (drive unit)
326 housings
339 characteristic adjustment parts
350 lens units
360 lens units
365 first liquids
366 second liquids
367 electrodes
The F frame of making a video recording
L laser slit light (laser of light, slit-shaped)
LL2 liquid lens (lens)
W workpiece (object)
Embodiment
Below, an embodiment of the invention are described with reference to accompanying drawing.
First, with reference to Fig. 1, the example that the integral body of the robot system of present embodiment forms is described.
As shown in Figure 1, the robot system 1 of present embodiment has accumulator 20(container), robot 100, conveyer 30, sensor unit 300(article detection device) and robot controller 200(controller).
Accumulator 20 is such as the container that is the case shape that formed by resin or metal etc., and is configured near the pedestal 21 of configuration robot 100.In the inside of this accumulator 20, put into arbitrarily the workpiece W(object of a plurality of detected objects) (at random).In addition, each workpiece W also can not put into such container such as accumulator 20 grade, also can load in the first-class such applicable mounting surface of pedestal 21.In addition, as each workpiece W and their shape, being not particularly limited, can be various workpiece and shape.Now, each workpiece W and their shape also can be consistent with each other or similar, also can be mutually different.But, in each figure, the shape simplification of each workpiece W is represented with ellipse.
Robot 100 carries out the handover operation (so-called with choosing in cabinet) that a plurality of workpiece W in accumulator 20 are kept successively and transferred.As robot 100, as long as can carry out above-mentioned handover operation, do not limit especially, for example, can also use vertical articulated robot or horizontal articulated robot (SCARA robot) etc.In this example, as robot 100, apply vertical articulated robot, robot 100 has base station 101 and the rotation of being fixed on applicable stationary plane (for example, base plate etc.) and is arranged on freely the arm 102 on base station 101.Arm 102 has a plurality of joints from base station 101 sides to the front contrary with base station.In this arm 102, be built-in with a plurality of servo motors (not shown) that drive respectively above-mentioned a plurality of joints.In addition, the front end of arm 102 be provided with can holding workpiece W holding device 103.
As holding device 103, so long as device that can holding workpiece W, do not limit especially, for example, can also use can by finger parts control (a kind of mode of maintenance) workpiece W grasping device, can adsorb by drivings such as air or electromagnetic forces the adsorbent equipment of (a kind of mode of maintenance) workpiece W etc.In this example, as holding device 103, application grasping device, holding device 103 has the pair of finger parts 103a that can control workpiece W.Pair of finger parts 103a is driven by the applicable actuator (not shown) being built in holding device 103, and by expanding and dwindling mutual interval and carry out on-off action.In addition, as holding device 103, be not limited to have the device of above-mentioned structure, for example, can also be with thering are a plurality of finger parts and the swing by these a plurality of finger parts carrys out grabbing device of grabbing workpiece W etc.
Such robot 100 is by utilizing the finger parts 103a of grasping device 103 to control successively singly a plurality of workpiece W in accumulator 20, and they are transferred and loaded to predefined mounting position in conveyer 30, thereby transfers operation.
Conveyer 30 is carried to the equipment relevant to subsequent processing loading in the locational workpiece W of predefined mounting.
Sensor unit 300, by light cross-section method, detects respectively a plurality of workpiece W in accumulator 20, and detection (also comprises three-dimensional position and attitude as these a plurality of workpiece W three-dimensional outer shape separately.Identical below) 3D shape.This sensor unit 300 supports by applicable support unit 50 to be positioned at the mode of the top of accumulator 20.About sensor unit 300, illustrate in greater detail below.
Robot controller 200 forms such as the computing machine by having arithmetical unit, memory storage, input media etc., and Bing Yu robot 100 and sensor unit 300 connect in the mode that can intercom mutually.The three-dimensional shape information of the 3D shape separately of a plurality of workpiece W in the expression accumulator 20 of this robot controller 200 based on being detected by sensor unit 300, carry out the action (on-off action of the finger parts 103a of for example, the driving in each joint of arm 102, grasping device 103 etc.) of control 100 integral body.About robot controller 200, illustrate in greater detail below.
Then,, with reference to Fig. 2~Fig. 5, an example of the formation of sensor unit 300 is described.
As shown in Fig. 2~Fig. 5, sensor unit 300 has laser scanner 320, video camera 310, is arranged at lens unit 350 and the sensor controller 330 of video camera 310.
Laser scanner 320 has LASER Light Source unit 320A, rotating mirror 322, motor 323(drive unit) and angle detector 324.In addition, rotating mirror 322 and motor 323 are corresponding with scanner section.In addition, LASER Light Source unit 320A has LASER Light Source 321(light source) and the lens unit 360 that is arranged at LASER Light Source 321, LASER Light Source 321 and lens unit 360 are incorporated in common housing 326 and form.Laser (the light of LASER Light Source 321 illumination slit shapes.Below, be called " laser slit light ") L.Lens unit 360 is arranged between LASER Light Source 321 and rotating mirror 322, and the laser slit light L that refraction is irradiated from LASER Light Source 321 makes its optically focused.About lens unit 360, illustrate in greater detail below.Rotating mirror 322 receives the laser slit light L that has passed through lens unit 360, to reflections such as workpiece W, and to projections such as workpiece W.Motor 323 makes rotating mirror 322 rotations, thereby the anglec of rotation of rotating mirror 322 is changed.By utilizing motor 323 to make rotating mirror 322 rotation, can make projection position on workpiece W in the laser slit light L Quan TT of view field etc. to (the direction of scanning of regulation of the direction shown in arrow A.Below, be called " direction of scanning A ") mobile, the full TT of view field is comprised of a plurality of view fields that become the projection objects of workpiece W etc.Angle detector 324 detects the anglec of rotation of rotating mirror 322.
When move to direction of scanning A as mentioned above at the projection position on the workpiece W of video camera 310 in the laser slit light L Quan TT of view field etc., with desirable frame per second (interval), the outward appearance of workpiece W in the full TT of view field at the projection position that comprises this and be moved etc. is made a video recording continuously.Particularly, video camera 310 from the view field of counter-scanning direction (reverse direction of the direction shown in the arrow A) side end of the TT of projection position Quan view field of laser slit light L generate until along during the view field of the direction of scanning A successively direction of scanning A side end of the mobile full TT of view field of arrival (, cross over the full TT of view field whole region during), make a video recording continuously.Then, video camera 310 outputs a plurality of shooting frame F(s reference described later Fig. 6 corresponding with above-mentioned continuous shooting).In Fig. 6, illustrate from an example of the shooting frame F of video camera 310 outputs.As shown in Figure 6, in shooting frame F, comprise the outward appearance of workpiece W in the full TT of view field etc. and the action at mobile, the laser slit light L projection position on this workpiece W etc. constantly as mentioned above.
Lens unit 350 is arranged between the projection position and the imaging apparatus 311 of video camera 310 of laser slit light L on above-mentioned workpiece W etc., and refraction makes its optically focused at the laser slit light L of projection position reflection.About lens unit 350, illustrate in greater detail below.
Sensor controller 330 forms such as the computing machine by having arithmetical unit and memory storage etc., and controls the action of sensor unit 300 integral body.A plurality of shooting frame Fs of this sensor controller 330 based on from video camera 310 outputs, detect respectively a plurality of workpiece W in accumulator 20, and detect these a plurality of workpiece W 3D shape separately.About sensor controller 330, illustrate in greater detail below.
Then,, with reference to Fig. 7, an example of the formation of lens unit 350,360 is described.
As shown in Figure 7, lens unit 350 has cylindrical vessel 354, and cylindrical vessel 354 is constituted as at the both ends of cylinder part 351 and engages the cover 352,353 with photopermeability.The first liquid 355 consisting of the non-electrolytic solution with insulativity and photopermeability and the second liquid 356 consisting of the electrolytic solution with electric conductivity and photopermeability are adjacent to enclose the inside of cylindrical vessel 354.First liquid 355 and second liquid 356 have non-miscibility mutually, and refractive index is mutually different.By these first liquids 355 and second liquid 356, form liquid lens LL1.Liquid lens LL1 is arranged between the projection position and the imaging apparatus 311 of video camera 310 of laser slit light L on above-mentioned workpiece W etc.In addition, for second liquid 356 being applied to the electrode 357 of the voltage of expectation, with ring-type, be formed on the inwall of cylindrical vessel 354.Above-mentioned first liquid 355 and second liquid 356 contact with this electrode 357 via insulator 358.
In such lens unit 350, when applying voltage between 359 pairs of second liquids 356 of the voltage bringing device by applicable and electrode 357, according to this, execute alive size, first liquid 355 changes with the curvature of the interface I F1 of second liquid 356.Thus, the refracting power of liquid lens LL1, be that the optical characteristics such as the focal length of liquid lens LL1 or visual angle change.
In addition, lens unit 360 is formations identical with said lens unit 350.That is, lens unit 360 has cylindrical vessel 364, and cylindrical vessel 364 is constituted as at the both ends of cylinder part 361 and engages the cover 362,363 with photopermeability.The first liquid 365 consisting of the non-electrolytic solution with insulativity and photopermeability and the second liquid 366 consisting of the electrolytic solution with electric conductivity and photopermeability are adjacent to enclose the inside of cylindrical vessel 364.First liquid 365 and second liquid 366 have non-miscibility mutually, and refractive index is mutually different.By these first liquids 365 and second liquid 366, form liquid lens LL2(lens).Liquid lens LL2 is arranged between LASER Light Source 321 and rotating mirror 322.In addition, for second liquid 366 being applied to the electrode 367 of the voltage of expectation, with ring-type, be formed on the inwall of cylindrical vessel 364.Above-mentioned first liquid 365 and second liquid 366 contact with this electrode 367 via insulator 368.
In such lens unit 360, when applying voltage between 369 pairs of second liquids 366 of the voltage bringing device by applicable and electrode 367, according to this, execute alive size, the curvature of the interface I F2 between first liquid 365 and second liquid 366 changes.Thus, the refracting power of liquid lens LL2, be that the optical characteristics such as the focal length of liquid lens LL2 or visual angle change.
Then,, with reference to Fig. 8, an example of the functional formation of sensor controller 330 is described.
As shown in Figure 8, sensor controller 330 has light source control portion 341, motor control portion 342, camera control portion 338, characteristic adjustment part 339, image acquiring unit 331, image storage part 332, range image generating unit 333, range image storage part 334, composograph generating unit 336 and test section 340.
Light source control portion 341 controls above-mentioned LASER Light Source 321, and irradiating laser slit light L.The rotation angle information of the above-mentioned rotating mirror 322 that 342 inputs of motor control portion detect by above-mentioned angle detector 324, the rotation angle information based on this input, controls said motor 323, and makes rotating mirror 322 rotations.Camera control portion 338 controls above-mentioned video camera 310, and at the projection position of laser slit light L, cross over the full TT of view field whole region during, repeatedly make a video recording continuously, and export corresponding a plurality of shooting frame F.
At this, when being carried out above-mentioned single pass of repeatedly making a video recording continuously by video camera 310, because the position of workpiece W etc. is different or it is concavo-convex etc. like that to exist on the surface of workpiece W etc., the projection position of laser slit light L on this workpiece W etc. exists concavo-convex sometimes.Now, according to projection position, be recess or protuberance, the optical path length of the laser slit light L on the surface (projection position) from LASER Light Source 321 to workpiece W etc. or different to the optical path length of the laser slit light L of the imaging apparatus 311 of video camera 310 from the surface (projection position) of workpiece W etc.Therefore, above-mentioned in the situation that, if establish the focal length of above-mentioned each lens unit 350,360 or visual angle etc. for fixed value, may be difficult to make a plurality of positions mutual different with the focal length of described each lens unit 350,360, the focal position of each lens unit 350,360 corresponding.Therefore, due to the difference (difference of the focal length of lens unit 360) of the optical path length of the surperficial laser slit light L from LASER Light Source 321 to workpiece W etc. or from the surface of workpiece W etc. to the difference (difference of the focal length of lens unit 350) of the optical path length of the laser slit light L of the imaging apparatus 311 of video camera 310, therefore sometimes may be difficult to carry out high-precision shooting.Therefore, in the present embodiment, the characteristic adjustment part 339 that setting can be adjusted optical characteristics such as the focal length of above-mentioned each lens unit 350,360 or visual angles.
When above-mentioned single pass, the optical characteristics of 339 pairs of above-mentioned each lens units 350,360 in characteristic adjustment part is adjusted.Particularly, characteristic adjustment part 339 in the situation that each liquid lens LL1 of not mobile each lens unit 350,360, the position of LL2 the optical characteristics of this each liquid lens LL1, LL2 is adjusted.; characteristic adjustment part 339 is by increasing and decreasing the voltage applying between the second liquid 366 of the voltage that applies between the second liquid 356 of 359 pairs of lens units 350 of above-mentioned voltage bringing device and electrode 357 and 369 pairs of lens units 360 of above-mentioned voltage bringing device and electrode 367; the curvature of above-mentioned each interface IF1, IF2 is changed, thereby the optical characteristics of each liquid lens LL1, LL2 is adjusted.
In the present embodiment, the mode of characteristic adjustment part 339 so that the focal position of each liquid lens LL1, LL2 is switched between the mutual different a plurality of positions of the focal length of this each liquid lens LL1, LL2, increases and decreases the voltage that each voltage bringing device 359,369 applies.In this example, as shown in Fig. 9 (a), as the mutual different a plurality of positions of focal length of liquid lens LL1, set upper side position P1a and these two positions of lower side position P1b.Upper side position P1a is the position (scope) that the focal length of liquid lens LL1 is relatively short short focal length.Lower side position P1b is the position (scope) that the focal length of liquid lens LL1 is the long focal length longer than above-mentioned short focal length.In addition, in this example, as shown in Figure 9 (b), as the mutual different a plurality of positions of focal length of liquid lens LL2, set upper side position P2a and these two positions of lower side position P2b.Upper side position P2a is the position (scope) that the focal length of liquid lens LL2 is relatively short short focal length.Lower side position P2b is the position (scope) that the focal length of liquid lens LL2 is the long focal length longer than above-mentioned short focal length.In addition, in Fig. 9 (b), for convenience of explanation, LASER Light Source 321, liquid lens LL2, rotating mirror 322 and workpiece W etc. are illustrated as and are located along the same line.
In addition, when making above-mentioned each upper side position P1a, P2a and primary importance at once, above-mentioned each short focal length is corresponding with the first focal length, and above-mentioned each lower side position P1b, P2b are corresponding with the second place, and above-mentioned each long focal length is corresponding with the second focal length.On the contrary, when making above-mentioned each lower side position P1b, P2b and primary importance at once, above-mentioned each long focal length is corresponding with the first focal length, and above-mentioned each upper side position P1a, P2a are corresponding with the second place, and above-mentioned each short focal length is corresponding with the second focal length.
And, characteristic adjustment part 339 is alternately switched to the focal position of each liquid lens LL1, LL2 the mode of above-mentioned each upper side position P1a, P2a and above-mentioned each lower side position P1b, P2b with switching timing corresponding to shooting timing with video camera 310, alternately increases and decreases the voltage that each voltage bringing device 359,369 applies.
For example, characteristic adjustment part 339 increases and decreases each voltage bringing device 359 in the following manner, 369 voltages that apply: in response to the n time in the switching timing repeatedly in single pass (n is positive odd number) switching timing (the 1st, 3,5,7 ... inferior switching timing), the focal position of each liquid lens LL1, LL2 is switched to each upper side position P1a, P2a, in response to the n+1 time switching timing (the 2nd, 4,6,8 ... inferior switching timing), the focal position of each liquid lens LL1, LL2 is switched to each lower side position P1b, P2b.In addition, below, the switch mode of the focal position of such each liquid lens LL1, LL2 is called to " the first switch mode ".Or, characteristic adjustment part 339 increases and decreases the voltage that each voltage bringing device 359,369 applies in the following manner: in response to the n time switching timing in the switching timing repeatedly in single pass, the focal position of each liquid lens LL1, LL2 is switched to each lower side position P1b, P2b, in response to the n+1 time switching timing, the focal position of each liquid lens LL1, LL2 is switched to each upper side position P1a, P2a.In addition, below, the switch mode of the focal position of such each liquid lens LL1, LL2 is called to " the second switch mode ".
In the present embodiment, characteristic adjustment part 339, alternately to repeat the mode of above-mentioned the first switch mode and above-mentioned the second switch mode in different in time Multiple-Scans, increases and decreases the voltage that each voltage bringing device 359,369 applies.
For example, characteristic adjustment part 339 increases and decreases the voltage that each voltage bringing device 359,369 applies in the following manner: in the N time (N is positive odd number) scanning (the 1st, 3,5,7 ... inferior scanning) in, in the mode corresponding with above-mentioned the first switch mode, switch the focal position of each liquid lens LL1, LL2, the N+1 time scanning the (the 2nd, 4,6,8 ... inferior scanning), in, in the mode corresponding with above-mentioned the second switch mode, switch the focal position of each liquid lens LL1, LL2.Or, characteristic adjustment part 339 also can increase and decrease the voltage that each voltage bringing device 359,369 applies in the following manner, in the scanning of the N time, in the mode corresponding with above-mentioned the second switch mode, switch the focal position of each liquid lens LL1, LL2, in the scanning of the N+1 time, in the mode corresponding with above-mentioned the first switch mode, switch the focal position of each liquid lens LL1, LL2.Wherein, below, for convenience of explanation, to characteristic adjustment part 339 with in the scanning of the N time, in the mode corresponding with above-mentioned the first switch mode, switch the focal position of each liquid lens LL1, LL2 and in the N+1 time scanning to switch with mode corresponding to above-mentioned the second switch mode the situation that the such mode in the focal position of each liquid lens LL1, LL2 increases and decreases each voltage bringing device 359,369 voltages that applied, describe.
In addition, in the present embodiment, above-mentioned camera control portion 338 so that in above-mentioned the N time scanning and above-mentioned the N+1 time scanning shooting start the mode that timing is staggered, control video camera 310.
Image acquiring unit 331 is obtained from a plurality of shooting frame F of above-mentioned video camera 310 outputs.A plurality of shooting frame F that obtain by image acquiring unit 331 are stored in image storage part 332.
Range image generating unit 333, for above-mentioned every 1 scanning, is used and is stored in a plurality of shooting frame F in this scanning in image storage part 332, according to the principle of triangulation, calculates the distance of video camera 310 and workpiece W etc.Particularly, the above-mentioned distance of the each several part on, that focus is consistent at projection position that represent, a plurality of laser slit light L in the frame of respectively making a video recording of range image generating unit 333 a plurality of shooting frame F of calculating in this scanning.Then, range image generating unit 333 be created in the each several part that the above-mentioned focus of appearance images of workpiece W in the full TT of view field etc. is consistent, comprise the above-mentioned range information calculating a range image DP(with reference to Figure 10 described later rear portion and Figure 11 rear portion).The range image DP generating by range image generating unit 333 is stored in range image storage part 334.
Figure 10 illustrates when the scanning of above-mentioned the N time from a plurality of shooting frame F of video camera 310 outputs and an example that passes through the range image DP of range image generating unit 333 generations.In addition, in Figure 10 top and Figure 11 described later top, for convenience of explanation, illustrate and do not put into accumulator 20 with a plurality of workpiece W but a plurality of (being two at this example) workpiece W corresponding a plurality of shooting frame F while loading 310 shooting of video camera under the state on pedestal 21.In addition, below, for convenience of explanation, M the shooting frame F corresponding with the M time (M be positive integer) shooting is called to " shooting frame FM ", M+1 the frame F that makes a video recording that makes a video recording corresponding with the M+1 time is called to " frame FM1 makes a video recording ".
As shown in Figure 10 top, show the action at the projection position of following laser slit light L: in M shooting frame FM when the N time scanning from a plurality of shooting frame F of video camera 310 outputs, on the upper surface of workpiece W within being present in above-mentioned each upper side position P1a, P2a, the focus of described laser slit light L is consistent, the lower surface (upper surface of pedestal 21) of workpiece W outside being present in this each upper side position P1a, P2a is upper, and the focus of described laser slit light L is inconsistent.On the other hand, show the action at the projection position of following laser slit light L: in M+1 shooting frame FM1, on the upper surface of workpiece W outside being present in above-mentioned each lower side position P1b, P2b, the focus of described laser slit light L is inconsistent, the lower surface (upper surface of pedestal 21) of workpiece W within being present in this each lower side position P1b, P2b is upper, and the focus of described laser slit light L is consistent.In addition, in Figure 10 top and Figure 11 described later top, with fine rule, represent the action at the projection position of the laser slit light L that focus is consistent, with thick line, represent the action at the projection position of the inconsistent laser slit light of focus L.In addition, as shown in the lower part of Figure 10, comprise range information in using the N time when scanning the each several part Pa that the above-mentioned focus of appearance images of range image DP that a plurality of shooting frame F shown in above-mentioned Figure 10 top generate two workpiece W on pedestal 21 is consistent.
In Figure 11, a plurality of shooting frame F that export from video camera 310 while being illustrated in above-mentioned the N+1 time scanning and an example that passes through the range image DP of range image generating unit 333 generations.
As shown in Figure 11 top, show the action at the projection position of following laser slit light L: in M shooting frame FM when the N+1 time scanning from a plurality of shooting frame F of video camera 310 outputs, on the upper surface of workpiece W within being present in above-mentioned each upper side position P1a, P2a, the focus of described laser slit light L is consistent, the lower surface (upper surface of pedestal 21) of workpiece W outside being present in this each upper side position P1a, P2a is upper, and the focus of laser slit light L is inconsistent.On the other hand, show the action at the projection position of following laser slit light L: in M+1 shooting frame FM1, on the upper surface of workpiece W outside being present in above-mentioned each lower side position P1b, P2b, the focus of described laser slit light L is inconsistent, the lower surface (upper surface of pedestal 21) of workpiece W within being present in this each lower side position P1b, P2b is upper, and the focus of described laser slit light L is consistent.In addition, as shown in the lower part of Figure 11, when the scanning of the N+1 time, use in the each several part Pb that the above-mentioned focus of appearance images of range image DP that a plurality of shooting frame F shown in above-mentioned Figure 11 top generate two workpiece W on pedestal 21 is consistent and comprise range information.
As shown in Figure 8, composograph generating unit 336 is when each end of scan, by be stored in range image storage part 334 mutually a plurality of (being two in this example) the range image DP in different repeatedly (in this example twice) scanning is synthetic in time, and generate composograph CP(with reference to Figure 12 described later).Composograph CP is the image for detection of the 3D shape separately of a plurality of workpiece W in accumulator 20.Particularly, composograph generating unit 336, when each end of scan, by the range image DP in up-to-date scanning with immediately the range image DP in the scanning before up-to-date scanning is synthetic, and generates composograph CP.That is, composograph generating unit 336 is synthetic by the part in the range image DP of the scanning before the part that comprises range information in the range image DP of the scanning based on up-to-date and the scanning based on immediately up-to-date.Thus, the part that composograph generating unit 336 immediately comprises range information in the range image DP of the scanning before up-to-date scanning by the part utilization that does not comprise range information in the range image DP of the scanning based on up-to-date based on this is supplemented.So, composograph generating unit 336 generates composograph CP.The composograph CP generating by composograph generating unit 336 is exported to test section 340.
Figure 12 represents by composograph generating unit 336 example of the synthetic composograph CP generating of range image DP, DP shown in above-mentioned Figure 10 bottom and Figure 11 bottom.
As shown in figure 12, range image DP, DP shown in above-mentioned Figure 10 bottom and Figure 11 bottom be synthesized two the workpiece Ws of generated composograph CP on pedestal 21 appearance images each part mentioned above Pa, Pb(, this appearance images roughly whole) in comprise range information.
As shown in Figure 8, test section 340, when each end of scan, obtains from the composograph CP of composograph generating unit 336 outputs.Then, test section 340 is used above-mentioned obtained composograph CP, the 3D shape separately of a plurality of workpiece W in detection accumulator 20.The detection signal of the testing result of expression based on test section 340 is exported to above-mentioned robot controller 200.
Thus, robot controller 200 obtains from the above-mentioned detection signal of test section 340 outputs.Then, robot controller 200, with the three-dimensional shape information based on representing with the above-mentioned detection signal being obtained, keeps and for example transfers a workpiece W(, is easy to the workpiece W that keeps etc. most) mode, make 100 actions of above-mentioned robot.
As mentioned above, in the sensor unit 300 of present embodiment, be provided with the characteristic adjustment part 339 that can adjust the optical characteristics that is arranged at each lens unit 350,360 in video camera 310 and LASER Light Source 321.The optical characteristics of each lens unit 350,360 can be adjusted in characteristic adjustment part 339 when single pass.Consequently, can be with above-mentioned workpiece W etc. surperficial accordingly concavo-convex, in above-mentioned single pass high speed ground switchable optics characteristic.Therefore, high-precision shooting can be carried out, the 3D shape of workpiece W can be detected accurately.
In addition, in the present embodiment, especially, be provided with the LASER Light Source 321 of irradiating laser slit light L, lens unit 360 has liquid lens LL2 between LASER Light Source 321 and rotating mirror 322.Thus, thereby at laser slit light L, scan, in structure that video camera 310 is made a video recording the 3D shape that detects workpiece W continuously, can carry out the surperficial concavo-convex corresponding high-precision shooting with above-mentioned workpiece W etc.Consequently, can detect accurately the 3D shape of workpiece W.
In addition, in the present embodiment, especially, LASER Light Source 321 and lens unit 360 are incorporated in common housing 326 and form LASER Light Source unit 320A.By using this LASER Light Source unit 320A, when being accommodated in to different housings, LASER Light Source 321 and lens unit 360 compare, and can be by sensor unit 300 miniaturizations.
In addition, in the present embodiment, especially, be provided with and receive the laser slit light L irradiating from LASER Light Source 321 and make it to the rotating mirror 322 of the reflections such as workpiece W and the anglec of rotation of rotating mirror 322 changed and make the Projection Division displacement moving motor 323 of laser slit light L on above-mentioned workpiece W etc.By using these rotating mirrors 322 and motor 323, can easily make to move to direction of scanning A to the laser slit light L of the reflections such as above-mentioned workpiece W.
In addition, in the present embodiment, especially, characteristic adjustment part 339 can in the situation that not the position of moving liquid lens LL2 adjust the optical characteristics of this liquid lens LL2.Thus, the situation of position that changes lens from driving mechanism by applicable etc. is different, not the position of moving liquid lens LL2 and carry out the surperficial concavo-convex corresponding high-precision shooting with above-mentioned workpiece W etc.In addition, can reduce for being provided for changing space and the expense of applicable driving mechanism etc. of the position of liquid lens LL2, therefore can make sensor unit 300 miniaturizations and cost degradation.
In addition, in the present embodiment, especially, liquid lens LL2 is set in lens unit 360, lens unit 360 has the electrode 367 that applies the voltage of expectation for the second liquid 366 to liquid lens LL2, and characteristic adjustment part 339 can increase and decrease the voltage applying between 360 pairs of second liquids 366 of voltage bringing device and electrode 367.Thus, utilization by the voltage swing to applying between second liquid 366 and electrode 367 come response good freely change the character of the liquid lens LL2 at focal length or visual angle etc., can in the situation that not the position of moving liquid lens LL2 carry out the surperficial concavo-convex corresponding high-precision shooting with above-mentioned workpiece W etc.
In addition, in the present embodiment, especially, characteristic adjustment part 339 is to give the mode of switching between a plurality of positions of mutually different focal lengths, the voltage applying between increase and decrease 360 pairs of second liquids 366 of voltage bringing device and electrode 367 by the focal position of liquid lens LL2.Thus, even in the situation that the optical path length of the surperficial laser slit light L causing from LASER Light Source 321 to above-mentioned workpiece W etc. due to the surperficial concavo-convex grade of above-mentioned workpiece W etc. increases or shortens, also can be correspondingly, in above-mentioned single pass, response is switched the focal length of liquid lens LL2 goodly at high speed, and can carry out high-precision shooting.
In addition, in the present embodiment, especially, characteristic adjustment part 339, the focal position of liquid lens LL2 is alternately switched to the mode of upper side position P2a and lower side position P2b, alternately increases and decreases the voltage applying between 360 pairs of second liquids 366 of voltage bringing device and electrode 367.Thus, during the protuberance projection of the above-mentioned workpiece W relatively shortening at the optical path length of the surperficial laser slit light L to from LASER Light Source 321 to above-mentioned workpiece W etc. etc., can be using the focal position of liquid lens LL2 as upper side position P2a, during the recess projection of the above-mentioned workpiece W relatively increasing at the optical path length of the surperficial laser slit light L to from LASER Light Source 321 to above-mentioned workpiece W etc. etc., can be using the focal position of liquid lens LL2 as lower side position P2b.Consequently, can carry out effectively high-precision shooting.
In addition, in the robot system 1 of present embodiment, by robot 100, keep successively and transfer a plurality of workpiece W in accumulator 20.In the present embodiment, when above-mentioned a plurality of workpiece W are transferred, the control based on robot controller 200, as described above, according to the 3D shape of each workpiece W being detected by sensor unit 300, makes robot 100 actions.Thus, can be smooth and easy and carry out effectively the handover of above-mentioned a plurality of workpiece W.In addition, because the 3D shape of each workpiece W is as described above detected accurately, therefore can successfully carry out by the handover of 100 couples of above-mentioned a plurality of workpiece W of robot effectively.In addition, as in the present embodiment, in sensor unit 300, can be in the situation that when the optical characteristics of this liquid lens LL2 not be adjusted in the position of moving liquid lens LL2 in characteristic adjustment part 339, can reduce for being provided for changing space and the expense of applicable driving mechanism etc. of the position of liquid lens LL2, therefore, can make sensor unit 300 miniaturizations and cost degradation.In addition, can make as described above the result of sensor unit 300 miniaturizations be, can avoid and the interference of object around.
In addition, embodiments of the present invention are not limited to foregoing, in the scope that does not depart from its purport and technological thought, can carry out various distortion.
; in the above-described embodiment; when increase and decrease between second liquid 356 and electrode 357, apply apply voltage adjust the focal length of liquid lens LL1 and increase and decrease between second liquid 366 and electrode 367, apply apply voltage and adjust the focal length of liquid lens LL2 time; characteristic adjustment part 339, the focal position of each liquid lens LL1, LL2 is alternately switched to the mode of each upper side position P1a, P2a and each lower side position P1b, P2b, alternately increases and decreases the above-mentioned voltage that respectively applies.Yet embodiments of the present invention are not limited to this.For example, can be also characteristic adjustment part 339 with by the focal position of each liquid lens LL1, LL2 giving the mode of switching between more than three position of mutually different focal lengths, increase and decrease the above-mentioned voltage that respectively applies.Or, previous shooting frame F(the second shooting frame that characteristic adjustment part 339 also can be based on this frame F that makes a video recording) image pickup result, set by 310 pairs one shooting frame F(the first shooting frame of video camera) the above-mentioned voltage that respectively applies while making a video recording.Now, the trickle adjustment of the focal length of each liquid lens LL1, LL2 can be carried out accurately, therefore high-precision shooting can be carried out more effectively.
In addition, in the above-described embodiment, sensor unit 300 is so that being positioned at the mode of the top of accumulator 20 supports by support unit 50, but embodiments of the present invention are not limited thereto.For example, sensor unit 300 also can be installed on the applicable position (for example, front of arm 102 etc.) of robot 100.Like this, even in the situation that sensor unit 300 is installed on to the enterprising enforcement use of robot 100, can make as described above the result of sensor unit 300 miniaturizations be, can avoid and the interference of object around.
In addition, in the above-described embodiment, as lens, each lens unit 350,360 has liquid lens LL1, LL2, Characteristics Control portion 339 is by increasing and decreasing the above-mentioned refracting power that voltage changes each liquid lens LL1, LL2 that respectively applies, thereby in the situation that not the position of mobile each lens adjusted the optical characteristics of these each lens.Yet embodiments of the present invention are not limited to this.For example, lens unit (for example has liquid lens lens in addition, the Zoom lens such as liquid crystal lens), Characteristics Control portion also can be by using applicable method to change the refracting power of lens, and in the situation that not the position of mobile lens adjust the optical characteristics of these lens.Or, can be also, lens unit has a plurality of lens and drives the applicable driving mechanism of these a plurality of lens, and Characteristics Control portion adjusts the position relationship of a plurality of lens by driving mechanism is driven, thereby adjusts the optical characteristics of lens unit.
In addition, in the above-described embodiment, LASER Light Source 321 and lens unit 360 are contained in common housing 326 and form LASER Light Source unit 320A, but embodiments of the present invention are not limited to this.For example, LASER Light Source 321 and lens unit 360 also can be accommodated in different housings.
In addition, in the above-described embodiment, as light source, use the LASER Light Source 321 of irradiating laser slit light L, but embodiments of the present invention are not limited to this.For example, as light source, also can use the applicable light source (for example, projector etc.) of the light such as irradiation mode light.
In addition, in the above-described embodiment, on LASER Light Source 321 and video camera 310 both sides, be provided with lens unit 350,360, but embodiments of the present invention are not limited to this.For example, also can only on the either party of LASER Light Source 321 and video camera 310, lens unit be set.
In addition, in the above-described embodiment, by utilizing motor 323 to make rotating mirror 322 rotation, the laser slit light L that irradiates is moved to the irradiated site of workpiece W etc., but embodiments of the present invention are not limited to this from LASER Light Source 321.For example, also can by change the light that irradiates from LASER Light Source 321 light sources such as grade towards or make light source itself mobile, the irradiated site moving from the light of light source irradiation to workpiece W etc.Now, change the light that irradiates from light source towards or the structure of mobile this light source itself corresponding with scanner section.
In addition, in the above-described embodiment, sensor unit 300 is applied to robot system 1, but sensor unit also can be applied to robot system system in addition.
In addition, the arrow shown in Fig. 8 is used for representing the example flowing of signal, and is not intended to limit the flow direction of signal.
In addition, except having narrated above, also the method based on above-mentioned embodiment etc. suitably can be combined to utilize.
In addition,, although do not enumerate, above-mentioned embodiment etc. can apply various changes and implement in the scope that does not depart from its purport.

Claims (11)

1. an article detection device, detects the object of detected object, it is characterized in that having:
Light source, described light source irradiates light;
Scanner section, described scanner section makes the described light irradiating from described light source move to the direction of scanning of regulation to the projection position of described project objects;
Video camera, described video camera, when described scanner section moves the described Projection Division displacement of described light on described object, is made a video recording with the interval of expectation continuously to the outward appearance of the described object at the projection position that comprises described movement;
Lens unit, described lens unit is arranged at described light source or described video camera; And
Characteristic adjustment part, the optical characteristics of described lens unit can be adjusted in described characteristic adjustment part when described video camera is made a video recording continuously.
2. article detection device as claimed in claim 1, is characterized in that,
Described light source is the LASER Light Source of the laser of illumination slit shape,
Described lens unit has the lens that are arranged between described LASER Light Source and described scanner section.
3. article detection device as claimed in claim 2, is characterized in that,
Described LASER Light Source and described lens unit are accommodated in common housing and form LASER Light Source unit.
4. article detection device as claimed in claim 2 or claim 3, is characterized in that,
Described scanner section has:
Rotating mirror, described rotating mirror receives the described laser irradiating from described LASER Light Source and it is reflected to described object; And
Drive unit, described drive unit changes the anglec of rotation of described rotating mirror, thereby makes the described Projection Division displacement of described laser on described object moving.
5. article detection device as claimed in claim 4, is characterized in that,
Described characteristic adjustment part is constituted as can be in the situation that do not move the optical characteristics that described lens are adjusted in the position of described lens.
6. article detection device as claimed in claim 5, is characterized in that,
Described lens are the liquid lenss that consist of the liquid of being enclosed,
Described lens unit has for the described liquid of described liquid lens being applied to the electrode of the voltage of expectation,
Described characteristic adjustment part is constituted as and can increases and decreases the voltage that described electrode is applied.
7. article detection device as claimed in claim 6, is characterized in that,
Described characteristic adjustment part is to give the mode of switching between a plurality of positions of mutually different focal lengths, the voltage that increase and decrease applies described electrode by the focal position of described liquid lens.
8. article detection device as claimed in claim 7, is characterized in that,
Described characteristic adjustment part, the focal position of described liquid lens is alternately switched to the mode of giving the primary importance of the first focal length and giving the second place of second focal length longer than the first focal length, alternately increases and decreases the voltage that described electrode is applied.
9. article detection device as claimed in claim 6, is characterized in that,
The previous shooting frame of described characteristic adjustment part based on the first shooting frame, the i.e. image pickup result of the second shooting frame, set the voltage that applies applying to described electrode when described video camera carries out the shooting of described the first shooting frame.
10. a robot system, is characterized in that, has:
Article detection device as claimed in any one of claims 1-9 wherein;
Container, puts into described container by a plurality of described objects;
Robot, described robot keeps successively and transfers the described a plurality of objects in described container; And
Controller, the 3D shape separately of the described a plurality of objects in the described container of described controller based on detecting by described article detection device makes described robot motion.
11. robot systems as claimed in claim 10, is characterized in that,
Described article detection device is installed in described robot.
CN201310740927.4A 2013-02-19 2013-12-26 Object detector and robot system Pending CN103994728A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2013-030144 2013-02-19
JP2013030144A JP2014159989A (en) 2013-02-19 2013-02-19 Object detector and robot system

Publications (1)

Publication Number Publication Date
CN103994728A true CN103994728A (en) 2014-08-20

Family

ID=51308962

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310740927.4A Pending CN103994728A (en) 2013-02-19 2013-12-26 Object detector and robot system

Country Status (2)

Country Link
JP (1) JP2014159989A (en)
CN (1) CN103994728A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108253229A (en) * 2017-12-29 2018-07-06 深圳市越疆科技有限公司 Control method, device and the terminal device of mechanical structure safe operation
CN110030962A (en) * 2018-01-11 2019-07-19 株式会社三丰 Lens measuring device and lens measurement method
CN110596115A (en) * 2018-06-12 2019-12-20 欧姆龙株式会社 Image inspection system and control method thereof
CN110645918A (en) * 2018-06-26 2020-01-03 精工爱普生株式会社 Three-dimensional measuring device, control device, and robot system
CN112008717A (en) * 2019-05-30 2020-12-01 松下i-PRO传感解决方案株式会社 Camera and robot system

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6401286B2 (en) * 2014-10-08 2018-10-10 株式会社Fuji Image acquisition apparatus and robot apparatus
WO2020084850A1 (en) * 2018-10-23 2020-04-30 パナソニックIpマネジメント株式会社 Light detection system
JP7219060B2 (en) * 2018-11-09 2023-02-07 株式会社キーエンス Displacement measuring device
CN112518126A (en) * 2020-12-14 2021-03-19 航天智造(上海)科技有限责任公司 Intelligent welding system based on large depth of field laser profiler

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002357408A (en) * 2001-03-25 2002-12-13 Omron Corp Optical measuring device
CN101144913A (en) * 2007-10-16 2008-03-19 东南大学 Three-dimensional stereo display
JP2010043954A (en) * 2008-08-12 2010-02-25 Tokyo Seimitsu Co Ltd Dimension measuring apparatus
JP2010164595A (en) * 2010-05-07 2010-07-29 Shin Gijutsu Consul:Kk Three-dimensional object image scanner
JP2010197112A (en) * 2009-02-24 2010-09-09 Keyence Corp Optical type displacement meter
CN102096126A (en) * 2011-01-07 2011-06-15 南京邮电大学 Ion liquid-based microfluidic varifocus lens
JP2011191086A (en) * 2010-03-12 2011-09-29 Omron Corp Image processing method and image processing device
JP2012093104A (en) * 2010-10-25 2012-05-17 Yaskawa Electric Corp Shape measuring device, robot system, and shape measuring method
CN102466825A (en) * 2010-11-16 2012-05-23 中国科学院兰州化学物理研究所 Electrowetting zoom lens based on ion liquid

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002357408A (en) * 2001-03-25 2002-12-13 Omron Corp Optical measuring device
CN101144913A (en) * 2007-10-16 2008-03-19 东南大学 Three-dimensional stereo display
JP2010043954A (en) * 2008-08-12 2010-02-25 Tokyo Seimitsu Co Ltd Dimension measuring apparatus
JP2010197112A (en) * 2009-02-24 2010-09-09 Keyence Corp Optical type displacement meter
JP2011191086A (en) * 2010-03-12 2011-09-29 Omron Corp Image processing method and image processing device
JP2010164595A (en) * 2010-05-07 2010-07-29 Shin Gijutsu Consul:Kk Three-dimensional object image scanner
JP2012093104A (en) * 2010-10-25 2012-05-17 Yaskawa Electric Corp Shape measuring device, robot system, and shape measuring method
CN102466825A (en) * 2010-11-16 2012-05-23 中国科学院兰州化学物理研究所 Electrowetting zoom lens based on ion liquid
CN102096126A (en) * 2011-01-07 2011-06-15 南京邮电大学 Ion liquid-based microfluidic varifocus lens

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108253229A (en) * 2017-12-29 2018-07-06 深圳市越疆科技有限公司 Control method, device and the terminal device of mechanical structure safe operation
CN110030962A (en) * 2018-01-11 2019-07-19 株式会社三丰 Lens measuring device and lens measurement method
CN110596115A (en) * 2018-06-12 2019-12-20 欧姆龙株式会社 Image inspection system and control method thereof
CN110596115B (en) * 2018-06-12 2022-03-18 欧姆龙株式会社 Image inspection system and control method thereof
CN110645918A (en) * 2018-06-26 2020-01-03 精工爱普生株式会社 Three-dimensional measuring device, control device, and robot system
CN110645918B (en) * 2018-06-26 2021-07-20 精工爱普生株式会社 Three-dimensional measuring device, control device, and robot system
CN112008717A (en) * 2019-05-30 2020-12-01 松下i-PRO传感解决方案株式会社 Camera and robot system
US11813740B2 (en) 2019-05-30 2023-11-14 i-PRO Co., Ltd. Camera and robot system

Also Published As

Publication number Publication date
JP2014159989A (en) 2014-09-04

Similar Documents

Publication Publication Date Title
CN103994728A (en) Object detector and robot system
US10606152B2 (en) Multi-aperture imaging device, imaging system and method for capturing an object area
US10771668B2 (en) Multi-aperture imaging device, imaging system and method for capturing an object area
US10869020B2 (en) Method for capturing images of a preferably structured surface of an object and device for image capture
JP6837841B2 (en) Autofocus for folding optics array cameras
US10901073B2 (en) Illumination for zoned time-of-flight imaging
CN103873751A (en) Three-dimensional panoramic scanning device and three-dimensional module generating method
Altschuler et al. Laser electro-optic system for rapid three-dimensional (3-D) topographic mapping of surfaces
JP2020122969A (en) Multi-aperture image capturing device, image capturing system, and method of providing multi-aperture image capturing device
US20150081080A1 (en) Three-dimensional printing system and method for three-dimensional printing
KR20210062064A (en) Camera assembly and terminal device
CN104061878A (en) Shape Measuring Apparatus
JPH0772445A (en) Three-dimensional display device
US9420235B2 (en) Measuring system for a 3D object
CN211123576U (en) Circuit for optical anti-shake driving device
JP2007517264A (en) Multidimensional imaging apparatus, system and method
JP2009544952A (en) Dynamic image recording with image creation sensor
JP2018516371A5 (en)
US20200208965A1 (en) Image pick-up device, image measurement apparatus, non-contact displacement-detecting device and non-contact profile-measuring device
JP2014228527A (en) Image measurement device
Wang et al. Dynamic depth-of-field projection for 3D projection mapping
CN105338246B (en) Lens devices and imaging device including the lens devices
KR101698449B1 (en) The projector module mounted to the mobile for the 3D scanning
CN207300540U (en) Optical detection device
US10642025B2 (en) Method for controlling position of a linear MEMS mirror with variable resolution and/or light intensity

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C02 Deemed withdrawal of patent application after publication (patent law 2001)
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20140820